2024-02-10T05:45:38.481 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T05:45:38.482 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} --name yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103 --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103/orig.config.yaml 2024-02-10T05:45:38.530 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T05:45:38.626 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103 branch: main description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} email: yweinste@redhat.com first_in_suite: false job_id: '7555103' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 1610 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi139.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDzupiDa8iCoDpVbGnF1CjaQKdhJ8u3hRFiv7+qev9jsLUM5KYuhRkDsNisU3v+38KyhuRC5TJmQviJeqfeSbc0= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGPhfqC7uuXe2tG6OI42hqF3SJGQ+KtDKznYuzD3pOZLwZUoxXUsmIM4CJWmLt6gwiQgxGYdMOHJ+knZoP4uV7c= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:44:16 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-10T05:45:38.627 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-10T05:45:38.628 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-10T05:45:38.628 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T05:45:38.629 INFO:teuthology.task.internal:Checking packages... 2024-02-10T05:45:38.656 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-10T05:45:38.656 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T05:45:38.657 INFO:teuthology.packaging:ref: None 2024-02-10T05:45:38.657 INFO:teuthology.packaging:tag: None 2024-02-10T05:45:38.657 INFO:teuthology.packaging:branch: main 2024-02-10T05:45:38.657 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:45:38.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-10T05:45:38.937 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-10T05:45:38.939 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T05:45:38.945 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T05:45:38.945 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T05:45:38.968 INFO:teuthology.task.internal:Saving configuration 2024-02-10T05:45:38.987 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T05:45:38.993 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:45:39.017 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:40:12.340760', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDzupiDa8iCoDpVbGnF1CjaQKdhJ8u3hRFiv7+qev9jsLUM5KYuhRkDsNisU3v+38KyhuRC5TJmQviJeqfeSbc0='} 2024-02-10T05:45:39.040 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:40:12.342075', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGPhfqC7uuXe2tG6OI42hqF3SJGQ+KtDKznYuzD3pOZLwZUoxXUsmIM4CJWmLt6gwiQgxGYdMOHJ+knZoP4uV7c='} 2024-02-10T05:45:39.040 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T05:45:39.046 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-02-10T05:45:39.046 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-02-10T05:45:39.046 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T05:45:39.120 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6fb086baf0>, signals=[15]) 2024-02-10T05:45:39.120 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T05:45:39.126 INFO:teuthology.task.internal:Opening connections... 2024-02-10T05:45:39.126 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2024-02-10T05:45:39.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:45:39.231 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-02-10T05:45:39.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:45:39.324 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T05:45:39.330 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2024-02-10T05:45:39.348 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2024-02-10T05:45:39.348 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2024-02-10T05:45:39.405 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Stream" 2024-02-10T05:45:39.405 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="9" 2024-02-10T05:45:39.405 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2024-02-10T05:45:39.406 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:45:39.406 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="9" 2024-02-10T05:45:39.406 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:45:39.406 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:45:39.406 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2024-02-10T05:45:39.406 INFO:teuthology.orchestra.run.smithi139.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:45:39.407 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:45:39.407 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:45:39.407 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:45:39.407 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:45:39.407 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:45:39.408 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2024-02-10T05:45:39.427 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-02-10T05:45:39.445 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-02-10T05:45:39.446 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-02-10T05:45:39.501 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-02-10T05:45:39.501 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:45:39.502 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:45:39.503 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-02-10T05:45:39.527 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T05:45:39.536 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T05:45:39.543 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T05:45:39.543 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:45:39.545 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:45:39.562 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T05:45:39.568 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T05:45:39.568 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:45:39.603 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:45:39.712 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T05:45:39.762 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T05:45:39.777 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T05:45:39.778 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T05:45:39.778 DEBUG:teuthology.orchestra.run.smithi139:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:45:39.778 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:45:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:39.795 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2024-02-10T05:45:39.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:39.796 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-02-10T05:45:39.851 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:45:39.852 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-361.el9.x86_64 2024-02-10T05:45:39.852 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2024-02-10T05:45:39.853 INFO:teuthology.orchestra.run.smithi139.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:45:39.853 INFO:teuthology.task.kernel:Running kernel on smithi139: 5.14.0-361.el9.x86_64 2024-02-10T05:45:39.853 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum install -y kernel 2024-02-10T05:45:42.214 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 9.3 MB/s | 6.7 MB 00:00 2024-02-10T05:45:42.337 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-9 - AppStream 7.9 MB/s | 6.7 MB 00:00 2024-02-10T05:45:43.998 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 2.8 MB/s | 2.0 MB 00:00 2024-02-10T05:45:44.187 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-9 - BaseOS 2.7 MB/s | 2.0 MB 00:00 2024-02-10T05:45:45.078 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-02-10T05:45:45.164 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-02-10T05:45:48.254 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - AppStream 9.0 MB/s | 19 MB 00:02 2024-02-10T05:45:53.454 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 5.9 MB 00:01 2024-02-10T05:45:55.738 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 15 kB 00:01 2024-02-10T05:45:58.874 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 20 MB 00:03 2024-02-10T05:46:03.939 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 6.1 kB/s | 1.7 kB 00:00 2024-02-10T05:46:06.270 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:46:06.270 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:46:06.355 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-02-10T05:46:06.367 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-02-10T05:46:06.367 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout:Install 4 Packages 2024-02-10T05:46:06.368 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T05:46:06.369 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 99 M 2024-02-10T05:46:06.369 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 122 M 2024-02-10T05:46:06.369 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-02-10T05:46:06.973 INFO:teuthology.orchestra.run.smithi139.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 9.9 MB/s | 6.0 MB 00:00 2024-02-10T05:46:08.032 INFO:teuthology.orchestra.run.smithi139.stdout:(2/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 24 MB/s | 39 MB 00:01 2024-02-10T05:46:08.232 INFO:teuthology.orchestra.run.smithi139.stdout:(3/4): kernel-core-5.14.0-419.el9.x86_64.rpm 11 MB/s | 21 MB 00:01 2024-02-10T05:46:08.623 INFO:teuthology.orchestra.run.smithi139.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 20 MB/s | 33 MB 00:01 2024-02-10T05:46:08.626 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:46:08.626 INFO:teuthology.orchestra.run.smithi139.stdout:Total 44 MB/s | 99 MB 00:02 2024-02-10T05:46:08.627 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-02-10T05:46:08.963 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-02-10T05:46:08.963 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-02-10T05:46:09.992 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-02-10T05:46:09.992 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-02-10T05:46:12.380 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-02-10T05:46:13.207 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:46:13.357 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:46:16.122 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:46:16.301 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:46:20.511 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 560 kB/s | 19 MB 00:34 2024-02-10T05:46:20.989 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:46:21.180 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:46:25.579 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 5.9 MB 00:01 2024-02-10T05:46:25.897 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:46:27.021 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2024-02-10T05:46:27.900 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 27 MB/s | 20 MB 00:00 2024-02-10T05:46:32.672 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-02-10T05:46:34.921 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:46:34.921 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:46:35.006 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-10T05:46:35.012 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-10T05:46:35.012 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout:Install 4 Packages 2024-02-10T05:46:35.013 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-10T05:46:35.014 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 99 M 2024-02-10T05:46:35.014 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 122 M 2024-02-10T05:46:35.014 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-10T05:46:35.349 INFO:teuthology.orchestra.run.smithi150.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 18 MB/s | 6.0 MB 00:00 2024-02-10T05:46:35.882 INFO:teuthology.orchestra.run.smithi150.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-02-10T05:46:36.449 INFO:teuthology.orchestra.run.smithi150.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 30 MB/s | 33 MB 00:01 2024-02-10T05:46:36.899 INFO:teuthology.orchestra.run.smithi150.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-02-10T05:46:36.902 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:46:36.903 INFO:teuthology.orchestra.run.smithi150.stdout:Total 53 MB/s | 99 MB 00:01 2024-02-10T05:46:36.903 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-10T05:46:37.289 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-10T05:46:37.289 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-10T05:46:38.346 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-10T05:46:38.346 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-10T05:46:40.779 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-10T05:46:41.427 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:46:41.575 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:46:44.369 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:46:44.551 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:46:49.317 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:46:49.530 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:46:54.288 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:46:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:46:55.057 INFO:teuthology.orchestra.run.smithi139.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:46:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T05:46:55.129 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:46:55.506 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:46:55.506 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:46:55.506 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:46:55.506 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:46:55.940 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:46:55.940 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T05:46:55.940 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-02-10T05:46:55.940 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:46:55.941 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:46:55.941 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:46:55.941 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:46:55.941 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T05:46:55.941 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-02-10T05:46:56.436 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:46:56.568 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:46:56.569 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:46:56.569 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:46:56.569 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:46:56.569 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:46:56.569 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-10T05:46:56.569 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi139.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:46:56.570 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:46:56.678 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:46:56.678 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:46:56.678 DEBUG:teuthology.orchestra.run.smithi139:> sudo rpm -qi grub2-tools 2024-02-10T05:46:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:Name : grub2-tools 2024-02-10T05:46:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:Epoch : 1 2024-02-10T05:46:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:Version : 2.06 2024-02-10T05:46:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:Release : 68.el9 2024-02-10T05:46:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: x86_64 2024-02-10T05:46:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:46:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:Group : Unspecified 2024-02-10T05:46:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:Size : 8264254 2024-02-10T05:46:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:License : GPLv3+ 2024-02-10T05:46:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:Packager : builder@centos.org 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:Vendor : CentOS 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:Summary : Support tools for GRUB. 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:Description : 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:hardware devices. 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T05:46:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:46:56.714 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:46:56.714 INFO:teuthology.task.kernel:Updating grub on smithi139 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:46:56.715 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:46:57.339 INFO:teuthology.orchestra.run.smithi139.stderr:Generating grub configuration file ... 2024-02-10T05:46:58.771 INFO:teuthology.orchestra.run.smithi139.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:46:58.819 INFO:teuthology.orchestra.run.smithi139.stderr:done 2024-02-10T05:46:58.821 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2024-02-10T05:46:58.837 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.3Y3AK5G0Jz 2024-02-10T05:46:58.837 DEBUG:teuthology.orchestra.run.smithi139:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3Y3AK5G0Jz 2024-02-10T05:46:58.906 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.3Y3AK5G0Jz 2024-02-10T05:46:59.040 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.3Y3AK5G0Jz is 6KB 2024-02-10T05:46:59.088 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.3Y3AK5G0Jz 2024-02-10T05:46:59.102 DEBUG:teuthology.orchestra.run.smithi139:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:46:59.166 INFO:teuthology.orchestra.run.smithi139.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:46:59.167 INFO:teuthology.orchestra.run.smithi139.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:46:59.167 INFO:teuthology.orchestra.run.smithi139.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:46:59.167 INFO:teuthology.orchestra.run.smithi139.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:46:59.168 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:46:59.320 DEBUG:teuthology.orchestra.run.smithi139:> sudo shutdown -r now 2024-02-10T05:47:23.637 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:47:23.637 INFO:teuthology.orchestra.run.smithi150.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:47:23.637 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-10T05:47:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:47:24.127 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:47:24.127 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:47:24.127 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:47:24.127 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:47:24.597 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:47:24.597 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-10T05:47:24.597 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-10T05:47:24.597 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:47:24.597 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:47:24.598 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:47:24.598 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:47:24.598 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-10T05:47:24.598 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-10T05:47:25.115 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:47:25.247 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:47:25.248 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:47:25.248 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:47:25.248 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:47:25.248 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:47:25.248 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-10T05:47:25.248 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:47:25.248 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:47:25.356 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:47:25.357 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:47:25.357 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2024-02-10T05:47:25.401 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2024-02-10T05:47:25.401 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2024-02-10T05:47:25.401 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.06 2024-02-10T05:47:25.401 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 68.el9 2024-02-10T05:47:25.401 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2024-02-10T05:47:25.401 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:47:25.401 INFO:teuthology.orchestra.run.smithi150.stdout:Group : Unspecified 2024-02-10T05:47:25.401 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 8264254 2024-02-10T05:47:25.401 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2024-02-10T05:47:25.401 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : builder@centos.org 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:47:25.402 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:47:25.403 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2024-02-10T05:47:25.403 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-10T05:47:25.403 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:47:25.404 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:47:25.404 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:47:25.404 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:47:26.003 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-02-10T05:47:27.306 INFO:teuthology.orchestra.run.smithi150.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:47:27.339 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-02-10T05:47:27.342 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-02-10T05:47:27.357 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.oO9gUHlDjj 2024-02-10T05:47:27.357 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oO9gUHlDjj 2024-02-10T05:47:27.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.oO9gUHlDjj 2024-02-10T05:47:27.569 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.oO9gUHlDjj is 6KB 2024-02-10T05:47:27.617 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.oO9gUHlDjj 2024-02-10T05:47:27.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:47:27.699 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:47:27.700 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:47:27.700 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:47:27.700 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:47:27.701 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:47:27.860 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-02-10T05:47:29.324 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-10T05:47:29.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-02-10T05:47:29.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:47:57.888 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-10T05:47:57.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-10T05:47:57.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:48:03.772 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-02-10T05:48:12.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-02-10T05:48:12.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:48:31.164 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-02-10T05:48:32.444 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-02-10T05:48:41.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-10T05:48:41.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:48:43.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-02-10T05:48:43.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:48:43.457 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-02-10T05:48:43.869 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-02-10T05:48:43.869 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:48:43.869 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2024-02-10T05:48:43.883 INFO:teuthology.orchestra.run.smithi139.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:48:43.883 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:48:43.883 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:48:43.883 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:48:44.884 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-10T05:48:44.884 DEBUG:teuthology.orchestra.run.smithi139:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:48:44.983 INFO:teuthology.orchestra.run.smithi139.stdout:ttyS1 2024-02-10T05:48:45.018 DEBUG:teuthology.parallel:result is None 2024-02-10T05:49:41.456 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T05:49:53.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-10T05:49:53.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:49:53.854 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-02-10T05:49:54.306 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-10T05:49:54.307 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:49:54.307 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-02-10T05:49:54.323 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:49:54.323 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:49:54.323 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:49:54.323 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:49:55.324 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-10T05:49:55.324 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:49:55.422 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2024-02-10T05:49:55.453 DEBUG:teuthology.parallel:result is None 2024-02-10T05:49:55.454 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T05:49:55.461 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T05:49:55.461 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:49:55.465 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:49:55.482 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T05:49:55.489 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T05:49:55.497 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T05:49:55.497 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:49:55.523 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:49:55.553 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T05:49:55.560 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T05:49:55.560 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:49:55.591 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:49:55.627 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:49:55.631 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:49:55.641 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:49:55.644 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:49:55.646 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T05:49:55.653 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T05:49:55.653 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:49:55.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:49:55.718 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T05:49:55.727 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T05:49:55.727 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:49:55.759 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:49:55.775 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:49:55.857 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:49:55.906 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:49:55.988 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:49:56.029 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-02-10T05:49:56.029 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:49:56.096 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:49:56.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:49:56.203 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:49:56.246 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:49:56.292 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-10T05:49:56.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:49:56.362 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2024-02-10T05:49:56.364 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-02-10T05:49:56.410 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:49:56.452 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:49:56.871 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T05:49:56.879 INFO:teuthology.task.internal:Starting timer... 2024-02-10T05:49:56.879 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T05:49:56.890 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T05:49:56.899 DEBUG:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2024-02-10T05:49:56.957 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: 2024-02-10T05:49:56.959 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-02-10T05:49:57.024 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-02-10T05:49:57.026 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T05:49:57.026 DEBUG:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2024-02-10T05:49:57.044 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2024-02-10T05:49:57.044 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-02-10T05:49:57.061 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-02-10T05:49:57.062 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi139.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive'} 2024-02-10T05:49:57.062 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:49:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:49:57.112 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:49:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:49:57.141 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T05:49:57.141 DEBUG:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:49:57.196 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:49:57.232 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T05:49:57.242 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-10T05:49:57.247 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-10T05:49:57.377 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T05:49:57.396 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T05:49:57.398 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi139.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T05:54:07.593 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi139.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2024-02-10T05:54:07.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-02-10T05:54:07.596 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:54:07.687 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-02-10T05:54:07.762 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-02-10T05:54:07.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-10T05:54:07.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:54:07.851 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-02-10T05:54:07.931 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-10T05:54:07.932 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T05:54:07.942 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T05:54:07.942 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T05:54:07.943 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:54:07.945 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T05:54:07.945 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:54:07.975 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T05:54:07.992 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T05:54:08.004 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T05:54:08.021 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T05:54:08.021 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2024-02-10T05:54:08.035 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2024-02-10T05:54:08.050 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-02-10T05:54:08.052 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T05:54:08.064 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-02-10T05:54:08.068 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T05:54:08.080 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T05:54:08.096 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T05:54:08.117 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:54:08.121 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:54:08.121 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-02-10T05:54:08.122 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:54:08.122 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:54:08.122 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:54:08.122 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:54:08.151 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:54:08.155 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:54:08.155 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-02-10T05:54:08.155 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:54:08.155 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:54:08.155 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:54:08.155 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:54:08.155 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T05:54:08.254 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T05:54:08.254 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-02-10T05:54:08.255 INFO:tasks.cephadm:Cluster fsid is ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:54:08.255 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T05:54:08.255 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.139', 'mon.c': '[v2:172.21.15.139:3301,v1:172.21.15.139:6790]', 'mon.b': '172.21.15.150'} 2024-02-10T05:54:08.255 INFO:tasks.cephadm:First mon is mon.a on smithi139 2024-02-10T05:54:08.255 INFO:tasks.cephadm:First mgr is y 2024-02-10T05:54:08.255 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T05:54:08.255 DEBUG:teuthology.orchestra.run.smithi139:> sudo hostname $(hostname -s) 2024-02-10T05:54:08.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname $(hostname -s) 2024-02-10T05:54:08.311 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-02-10T05:54:08.311 DEBUG:teuthology.orchestra.run.smithi139:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T05:54:08.586 INFO:teuthology.orchestra.run.smithi139.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 10 05:54 /home/ubuntu/cephtest/cephadm 2024-02-10T05:54:08.587 DEBUG:teuthology.orchestra.run.smithi150:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T05:54:08.746 INFO:teuthology.orchestra.run.smithi150.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 10 05:54 /home/ubuntu/cephtest/cephadm 2024-02-10T05:54:08.746 DEBUG:teuthology.orchestra.run.smithi139:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T05:54:08.761 DEBUG:teuthology.orchestra.run.smithi150:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T05:54:08.787 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-02-10T05:54:08.787 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-10T05:54:08.804 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-10T05:54:09.679 INFO:teuthology.orchestra.run.smithi139.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T05:54:09.719 INFO:teuthology.orchestra.run.smithi150.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T05:54:25.776 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-10T05:54:25.777 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-10T05:54:25.777 INFO:teuthology.orchestra.run.smithi150.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-10T05:54:25.777 INFO:teuthology.orchestra.run.smithi150.stdout: "repo_digests": [ 2024-02-10T05:54:25.777 INFO:teuthology.orchestra.run.smithi150.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-10T05:54:25.777 INFO:teuthology.orchestra.run.smithi150.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-10T05:54:25.777 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-10T05:54:25.777 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-10T05:54:27.833 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T05:54:27.833 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-10T05:54:27.833 INFO:teuthology.orchestra.run.smithi139.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-10T05:54:27.833 INFO:teuthology.orchestra.run.smithi139.stdout: "repo_digests": [ 2024-02-10T05:54:27.833 INFO:teuthology.orchestra.run.smithi139.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-10T05:54:27.833 INFO:teuthology.orchestra.run.smithi139.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-10T05:54:27.833 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-02-10T05:54:27.833 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T05:54:27.851 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph 2024-02-10T05:54:27.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph 2024-02-10T05:54:27.925 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /etc/ceph 2024-02-10T05:54:27.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /etc/ceph 2024-02-10T05:54:27.994 INFO:tasks.cephadm:Writing seed config... 2024-02-10T05:54:27.996 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-02-10T05:54:27.996 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T05:54:27.996 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T05:54:27.996 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-02-10T05:54:27.996 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T05:54:27.996 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T05:54:27.996 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T05:54:27.996 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T05:54:27.996 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T05:54:27.997 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-02-10T05:54:27.997 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T05:54:28.011 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ce44e742-c7d8-11ee-95b8-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-10T05:54:28.012 DEBUG:teuthology.orchestra.run.smithi139:mon.a> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.a.service 2024-02-10T05:54:28.053 DEBUG:teuthology.orchestra.run.smithi139:mgr.y> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.y.service 2024-02-10T05:54:28.096 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T05:54:28.096 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.139 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:54:28.294 INFO:teuthology.orchestra.run.smithi139.stderr:-------------------------------------------------------------------------------- 2024-02-10T05:54:28.294 INFO:teuthology.orchestra.run.smithi139.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'ce44e742-c7d8-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.139', '--skip-admin-label'] 2024-02-10T05:54:28.317 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: 4.9.3-dev 2024-02-10T05:54:28.319 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying podman|docker is present... 2024-02-10T05:54:28.341 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: 4.9.3-dev 2024-02-10T05:54:28.343 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying lvm2 is present... 2024-02-10T05:54:28.343 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying time synchronization is in place... 2024-02-10T05:54:28.349 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T05:54:28.355 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2024-02-10T05:54:28.362 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: enabled 2024-02-10T05:54:28.368 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: active 2024-02-10T05:54:28.368 INFO:teuthology.orchestra.run.smithi139.stderr:Unit chronyd.service is enabled and running 2024-02-10T05:54:28.368 INFO:teuthology.orchestra.run.smithi139.stderr:Repeating the final host check... 2024-02-10T05:54:28.389 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: 4.9.3-dev 2024-02-10T05:54:28.392 INFO:teuthology.orchestra.run.smithi139.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-10T05:54:28.392 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl is present 2024-02-10T05:54:28.392 INFO:teuthology.orchestra.run.smithi139.stderr:lvcreate is present 2024-02-10T05:54:28.397 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T05:54:28.403 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2024-02-10T05:54:28.410 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: enabled 2024-02-10T05:54:28.416 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: active 2024-02-10T05:54:28.417 INFO:teuthology.orchestra.run.smithi139.stderr:Unit chronyd.service is enabled and running 2024-02-10T05:54:28.417 INFO:teuthology.orchestra.run.smithi139.stderr:Host looks OK 2024-02-10T05:54:28.417 INFO:teuthology.orchestra.run.smithi139.stderr:Cluster fsid: ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:54:28.417 INFO:teuthology.orchestra.run.smithi139.stderr:Acquiring lock 140480674090912 on /run/cephadm/ce44e742-c7d8-11ee-95b8-87774f69a715.lock 2024-02-10T05:54:28.417 INFO:teuthology.orchestra.run.smithi139.stderr:Lock 140480674090912 acquired on /run/cephadm/ce44e742-c7d8-11ee-95b8-87774f69a715.lock 2024-02-10T05:54:28.418 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying IP 172.21.15.139 port 3300 ... 2024-02-10T05:54:28.418 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying IP 172.21.15.139 port 6789 ... 2024-02-10T05:54:28.418 INFO:teuthology.orchestra.run.smithi139.stderr:Base mon IP is 172.21.15.139, final addrv is [v2:172.21.15.139:3300,v1:172.21.15.139:6789] 2024-02-10T05:54:28.420 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.139 metric 100 2024-02-10T05:54:28.420 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.139 metric 100 2024-02-10T05:54:28.423 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T05:54:28.423 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-10T05:54:28.423 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1770sec hoplimit 64 pref medium 2024-02-10T05:54:28.425 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T05:54:28.425 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-10T05:54:28.425 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T05:54:28.425 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-10T05:54:28.426 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7f59/64 scope link 2024-02-10T05:54:28.426 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T05:54:28.426 INFO:teuthology.orchestra.run.smithi139.stderr:Mon IP `172.21.15.139` is in CIDR network `172.21.0.0/20` 2024-02-10T05:54:28.426 INFO:teuthology.orchestra.run.smithi139.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T05:54:28.427 INFO:teuthology.orchestra.run.smithi139.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T05:54:28.526 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-02-10T05:54:28.914 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Getting image source signatures 2024-02-10T05:54:28.914 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-02-10T05:54:28.914 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-02-10T05:54:28.914 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-02-10T05:54:28.915 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-02-10T05:54:28.915 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-02-10T05:54:28.915 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-10T05:54:29.044 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Writing manifest to image destination 2024-02-10T05:54:29.147 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-10T05:54:29.522 INFO:teuthology.orchestra.run.smithi139.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-10T05:54:30.120 INFO:teuthology.orchestra.run.smithi139.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-10T05:54:30.120 INFO:teuthology.orchestra.run.smithi139.stderr:Extracting ceph user uid/gid from container image... 2024-02-10T05:54:30.578 INFO:teuthology.orchestra.run.smithi139.stderr:stat: 167 167 2024-02-10T05:54:31.174 INFO:teuthology.orchestra.run.smithi139.stderr:Creating initial keys... 2024-02-10T05:54:31.574 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-authtool: AQCXD8dl9PwnHRAARYk5ZcBbA2aDWqWHPhPMaA== 2024-02-10T05:54:32.479 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-authtool: AQCYD8dl2074GBAAw9MB9AcJVlKHaCKogIaX4A== 2024-02-10T05:54:33.534 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-authtool: AQCZD8dlWeXKGhAAan++Z8Lsl/R6EkwRnkTxGA== 2024-02-10T05:54:34.015 INFO:teuthology.orchestra.run.smithi139.stderr:Creating initial monmap... 2024-02-10T05:54:34.415 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T05:54:34.415 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-02-10T05:54:34.415 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:54:34.415 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T05:54:35.030 INFO:teuthology.orchestra.run.smithi139.stderr:monmaptool for a [v2:172.21.15.139:3300,v1:172.21.15.139:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T05:54:35.030 INFO:teuthology.orchestra.run.smithi139.stderr:setting min_mon_release = octopus 2024-02-10T05:54:35.030 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/monmaptool: set fsid to ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:54:35.030 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T05:54:35.030 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-02-10T05:54:35.030 INFO:teuthology.orchestra.run.smithi139.stderr:Creating mon... 2024-02-10T05:54:35.918 INFO:teuthology.orchestra.run.smithi139.stderr:create mon.a on 2024-02-10T05:54:36.139 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T05:54:36.294 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ce44e742-c7d8-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-ce44e742-c7d8-11ee-95b8-87774f69a715.target. 2024-02-10T05:54:36.294 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-ce44e742-c7d8-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-ce44e742-c7d8-11ee-95b8-87774f69a715.target. 2024-02-10T05:54:36.599 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Failed to reset failed state of unit ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.a.service: Unit ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.a.service not loaded. 2024-02-10T05:54:36.606 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Created symlink /etc/systemd/system/ceph-ce44e742-c7d8-11ee-95b8-87774f69a715.target.wants/ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@.service. 2024-02-10T05:54:37.290 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: disabled 2024-02-10T05:54:37.296 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2024-02-10T05:54:37.297 INFO:teuthology.orchestra.run.smithi139.stderr:firewalld.service is not enabled 2024-02-10T05:54:37.297 INFO:teuthology.orchestra.run.smithi139.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T05:54:37.297 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mon to start... 2024-02-10T05:54:37.297 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mon... 2024-02-10T05:54:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:37 smithi139 systemd[1]: Started Ceph mon.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: cluster: 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: id: ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: services: 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.619438s) 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: data: 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-10T05:54:37.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-10T05:54:37.864 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: pgs: 2024-02-10T05:54:37.864 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:54:38.394 INFO:teuthology.orchestra.run.smithi139.stderr:mon is available 2024-02-10T05:54:38.394 INFO:teuthology.orchestra.run.smithi139.stderr:Assimilating anything we can from ceph.conf... 2024-02-10T05:54:38.509 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:38 smithi139 ceph-mon[24386]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T05:54:38.509 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:38 smithi139 ceph-mon[24386]: monmap e1: 1 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} 2024-02-10T05:54:38.509 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:38 smithi139 ceph-mon[24386]: fsmap 2024-02-10T05:54:38.509 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:38 smithi139 ceph-mon[24386]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T05:54:38.509 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:38 smithi139 ceph-mon[24386]: mgrmap e1: no daemons active 2024-02-10T05:54:38.509 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:38 smithi139 ceph-mon[24386]: from='client.? 172.21.15.139:0/1493860646' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T05:54:38.839 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:54:38.839 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: [global] 2024-02-10T05:54:38.839 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: fsid = ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:54:38.839 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.139:3300,v1:172.21.15.139:6789] 2024-02-10T05:54:38.839 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T05:54:38.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T05:54:38.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T05:54:38.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T05:54:38.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:54:38.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: [mgr] 2024-02-10T05:54:38.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-02-10T05:54:38.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T05:54:38.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:54:38.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: [osd] 2024-02-10T05:54:38.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T05:54:38.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T05:54:39.283 INFO:teuthology.orchestra.run.smithi139.stderr:Generating new minimal ceph.conf... 2024-02-10T05:54:39.508 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:39 smithi139 ceph-mon[24386]: from='client.? 172.21.15.139:0/3242752849' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T05:54:39.508 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:39 smithi139 ceph-mon[24386]: from='client.? 172.21.15.139:0/3242752849' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T05:54:40.181 INFO:teuthology.orchestra.run.smithi139.stderr:Restarting the monitor... 2024-02-10T05:54:40.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:40 smithi139 systemd[1]: Stopping Ceph mon.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:54:40.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:40 smithi139 bash[24603]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon.a" found: no such container 2024-02-10T05:54:40.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:40 smithi139 ceph-mon[24386]: from='client.? 172.21.15.139:0/3992545227' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:41.257 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:41 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a[24382]: 2024-02-10T05:54:41.001+0000 7f9a97370700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:54:41.257 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:41 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a[24382]: 2024-02-10T05:54:41.001+0000 7f9a97370700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T05:54:41.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:41 smithi139 podman[24616]: 2024-02-10 05:54:41.257529399 +0000 UTC m=+1.021932890 container died 793c0c50df383465199e984bfd1cce47a8bec1193d8e03b4baedd12664f94d2b (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, io.openshift.tags=base centos centos-stream, vcs-type=git, name=centos-stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, version=8, io.openshift.expose-services=, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., RELEASE=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-10T05:54:45.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:45 smithi139 podman[24616]: 2024-02-10 05:54:45.00137062 +0000 UTC m=+4.765774112 container cleanup 793c0c50df383465199e984bfd1cce47a8bec1193d8e03b4baedd12664f94d2b (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, release=754, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , architecture=x86_64, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, RELEASE=HEAD, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-10T05:54:45.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:45 smithi139 bash[24616]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a 2024-02-10T05:54:45.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:45 smithi139 bash[24644]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon.a" found: no such container 2024-02-10T05:54:47.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:46 smithi139 podman[24630]: 2024-02-10 05:54:46.643943891 +0000 UTC m=+5.375580741 container remove 793c0c50df383465199e984bfd1cce47a8bec1193d8e03b4baedd12664f94d2b (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, release=754, distribution-scope=public, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, vcs-type=git, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD) 2024-02-10T05:54:47.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:46 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-10T05:54:47.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:46 smithi139 systemd[1]: Stopped Ceph mon.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:54:47.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:46 smithi139 systemd[1]: Starting Ceph mon.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:54:47.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:46 smithi139 podman[24709]: 2024-02-10 05:54:46.787560977 +0000 UTC m=+0.020722022 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-02-10T05:54:48.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 podman[24709]: 2024-02-10 05:54:48.023846143 +0000 UTC m=+1.257007174 container create 2a614b8dfab775853ea8dab65e69d211335c9a1ad1656141abd37faa8e909f06 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, distribution-scope=public, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-10T05:54:48.779 INFO:teuthology.orchestra.run.smithi139.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-10T05:54:48.886 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 podman[24709]: 2024-02-10 05:54:48.70180993 +0000 UTC m=+1.934970961 container init 2a614b8dfab775853ea8dab65e69d211335c9a1ad1656141abd37faa8e909f06 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, distribution-scope=public, GIT_CLEAN=True, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, RELEASE=HEAD, version=8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., vcs-type=git, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, ceph=True, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream) 2024-02-10T05:54:48.886 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 podman[24709]: 2024-02-10 05:54:48.704966685 +0000 UTC m=+1.938127720 container start 2a614b8dfab775853ea8dab65e69d211335c9a1ad1656141abd37faa8e909f06 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, distribution-scope=public, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, GIT_CLEAN=True, ceph=True, io.openshift.expose-services=, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, name=centos-stream, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, build-date=2022-05-03T08:36:31.336870) 2024-02-10T05:54:48.886 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:54:48.887 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-10T05:54:48.887 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: pidfile_write: ignore empty --pid-file 2024-02-10T05:54:48.887 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: load: jerasure load: lrc 2024-02-10T05:54:48.887 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: RocksDB version: 6.15.5 2024-02-10T05:54:48.887 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T05:54:48.887 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Compile date Apr 18 2022 2024-02-10T05:54:48.887 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: DB SUMMARY 2024-02-10T05:54:48.888 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: DB Session ID: 6ZIF6IDRWLACU644PRHM 2024-02-10T05:54:48.888 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:54:48.888 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:54:48.888 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-10T05:54:48.888 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-10T05:54:48.888 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82663 ; 2024-02-10T05:54:48.888 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:54:48.888 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:54:48.889 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:54:48.889 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:54:48.889 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.env: 0x5649b36c6860 2024-02-10T05:54:48.889 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.fs: Posix File System 2024-02-10T05:54:48.889 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.info_log: 0x5649b3fd7dc0 2024-02-10T05:54:48.889 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:54:48.889 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.statistics: (nil) 2024-02-10T05:54:48.889 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.use_fsync: 0 2024-02-10T05:54:48.890 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:54:48.890 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:54:48.890 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:54:48.890 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:54:48.890 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:54:48.890 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:54:48.890 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.db_log_dir: 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:54:48.891 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:54:48.892 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:54:48.893 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:54:48.893 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.write_buffer_manager: 0x5649b40c8240 2024-02-10T05:54:48.893 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:54:48.893 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T05:54:48.893 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:54:48.893 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:54:48.893 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:54:48.893 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:54:48.893 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:54:48.893 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.unordered_write: 0 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.row_cache: None 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.wal_filter: None 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.preserve_deletes: 0 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:54:48.894 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:54:48.895 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_open_files: -1 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Compression algorithms supported: 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:54:48.896 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: kZSTD supported: 0 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.merge_operator: 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_filter: None 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:54:48.897 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5649b3fa3d00) 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout: index_type: 0 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout: data_block_index_type: 0 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout: index_shortening: 1 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout: hash_index_allow_collision: 1 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout: checksum: 1 2024-02-10T05:54:48.898 INFO:journalctl@ceph.mon.a.smithi139.stdout: no_block_cache: 0 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_cache: 0x5649b400e170 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_cache_options: 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: capacity : 536870912 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: num_shard_bits : 4 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: strict_capacity_limit : 0 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_cache_compressed: (nil) 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: persistent_cache: (nil) 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_size: 4096 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_size_deviation: 10 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_restart_interval: 16 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: index_block_restart_interval: 1 2024-02-10T05:54:48.899 INFO:journalctl@ceph.mon.a.smithi139.stdout: metadata_block_size: 4096 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout: partition_filters: 0 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout: use_delta_encoding: 1 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout: whole_key_filtering: 1 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout: verify_compression: 0 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout: format_version: 4 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout: enable_index_compression: 1 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_align: 0 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:54:48.900 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compression: NoCompression 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.num_levels: 7 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:54:48.901 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:54:48.902 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:54:48.902 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:54:48.902 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:54:48.902 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:54:48.902 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:54:48.902 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:54:48.902 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:54:48.902 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:54:48.902 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:54:48.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:54:48.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:54:48.905 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.ttl: 2592000 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544488732650, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-10T05:54:48.906 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544488733404, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79213, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77476, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71785, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707544488, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "eacfdd00-41a0-4e2a-befa-61a7562c02be", "db_session_id": "6ZIF6IDRWLACU644PRHM"}} 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544488733783, "job": 1, "event": "recovery_finished"} 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5649b3ff4a80 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: DB pointer 0x5649b4004000 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout: ** DB Stats ** 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:54:48.907 INFO:journalctl@ceph.mon.a.smithi139.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: L0 2/0 79.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.0 0.00 0.00 1 0.001 0 0 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: Sum 2/0 79.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.0 0.00 0.00 1 0.001 0 0 2024-02-10T05:54:48.908 INFO:journalctl@ceph.mon.a.smithi139.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.0 0.00 0.00 1 0.001 0 0 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.0 0.00 0.00 1 0.001 0 0 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: Cumulative compaction: 0.00 GB write, 23.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:54:48.909 INFO:journalctl@ceph.mon.a.smithi139.stdout: Interval compaction: 0.00 GB write, 23.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: L0 2/0 79.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.0 0.00 0.00 1 0.001 0 0 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: Sum 2/0 79.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.0 0.00 0.00 1 0.001 0 0 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:54:48.910 INFO:journalctl@ceph.mon.a.smithi139.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.0 0.00 0.00 1 0.001 0 0 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout: Cumulative compaction: 0.00 GB write, 23.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: starting mon.a rank 0 at public addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] at bind addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:54:48.911 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: mon.a@-1(???) e1 preinit fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: mon.a@-1(???).mds e1 new map 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: mon.a@-1(???).mds e1 print_map 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout: e1 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout: legacy client fscid: -1 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout: No filesystems configured 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:54:48.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T05:54:48.913 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 bash[24709]: 2a614b8dfab775853ea8dab65e69d211335c9a1ad1656141abd37faa8e909f06 2024-02-10T05:54:48.913 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 systemd[1]: Started Ceph mon.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:54:48.913 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T05:54:48.913 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: monmap e1: 1 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} 2024-02-10T05:54:48.913 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: fsmap 2024-02-10T05:54:48.913 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T05:54:48.913 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:48 smithi139 ceph-mon[24727]: mgrmap e1: no daemons active 2024-02-10T05:54:49.902 INFO:teuthology.orchestra.run.smithi139.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-10T05:54:49.902 INFO:teuthology.orchestra.run.smithi139.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:54:49.902 INFO:teuthology.orchestra.run.smithi139.stderr:Creating mgr... 2024-02-10T05:54:49.902 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying port 9283 ... 2024-02-10T05:54:50.085 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Failed to reset failed state of unit ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.y.service: Unit ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.y.service not loaded. 2024-02-10T05:54:50.092 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Created symlink /etc/systemd/system/ceph-ce44e742-c7d8-11ee-95b8-87774f69a715.target.wants/ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@.service. 2024-02-10T05:54:50.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:50 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1249721856' entity='client.admin' 2024-02-10T05:54:50.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:50 smithi139 podman[24954]: 2024-02-10 05:54:50.500322272 +0000 UTC m=+0.120785204 container create 5d5f244aa01fc4fc3432eb8aa01d4788e2a4b9f9d696caed15701c36a02717fc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , version=8, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, RELEASE=HEAD, io.openshift.expose-services=, GIT_CLEAN=True) 2024-02-10T05:54:50.840 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: disabled 2024-02-10T05:54:50.847 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2024-02-10T05:54:50.848 INFO:teuthology.orchestra.run.smithi139.stderr:firewalld.service is not enabled 2024-02-10T05:54:50.849 INFO:teuthology.orchestra.run.smithi139.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T05:54:50.856 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: disabled 2024-02-10T05:54:50.863 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2024-02-10T05:54:50.864 INFO:teuthology.orchestra.run.smithi139.stderr:firewalld.service is not enabled 2024-02-10T05:54:50.865 INFO:teuthology.orchestra.run.smithi139.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-10T05:54:50.865 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mgr to start... 2024-02-10T05:54:50.865 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mgr... 2024-02-10T05:54:50.991 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:50 smithi139 podman[24954]: 2024-02-10 05:54:50.727768743 +0000 UTC m=+0.348231675 container init 5d5f244aa01fc4fc3432eb8aa01d4788e2a4b9f9d696caed15701c36a02717fc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, name=centos-stream, ceph=True, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , release=754) 2024-02-10T05:54:50.991 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:50 smithi139 podman[24954]: 2024-02-10 05:54:50.73027101 +0000 UTC m=+0.350733943 container start 5d5f244aa01fc4fc3432eb8aa01d4788e2a4b9f9d696caed15701c36a02717fc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, architecture=x86_64, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, version=8, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, name=centos-stream) 2024-02-10T05:54:50.991 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:50 smithi139 bash[24954]: 5d5f244aa01fc4fc3432eb8aa01d4788e2a4b9f9d696caed15701c36a02717fc 2024-02-10T05:54:50.991 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:50 smithi139 systemd[1]: Started Ceph mgr.y for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:54:51.255 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:51 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:51.224+0000 7fe0d39a7000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:54:51.450 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:54:51.450 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2024-02-10T05:54:51.450 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsid": "ce44e742-c7d8-11ee-95b8-87774f69a715", 2024-02-10T05:54:51.450 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "health": { 2024-02-10T05:54:51.450 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T05:54:51.450 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T05:54:51.450 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T05:54:51.450 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:51.450 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T05:54:51.450 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 0 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "a" 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:51.451 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T05:54:51.452 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T05:54:51.452 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T05:54:51.452 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T05:54:51.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T05:54:51.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T05:54:51.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:51.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T05:54:51.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T05:54:51.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:51.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "available": false, 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "iostat", 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "nfs", 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "restful" 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:51.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modified": "2024-02-10T05:54:37.242795+0000", 2024-02-10T05:54:51.459 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:54:51.459 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:51.459 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T05:54:51.459 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2024-02-10T05:54:51.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:51 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3214027811' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:54:51.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:51 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:51.315+0000 7fe0d39a7000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:54:52.002 INFO:teuthology.orchestra.run.smithi139.stderr:mgr not available, waiting (1/15)... 2024-02-10T05:54:52.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:51 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:51.921+0000 7fe0d39a7000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:54:52.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:52.009+0000 7fe0d39a7000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:54:52.419 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:52.090+0000 7fe0d39a7000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:54:52.419 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:52.235+0000 7fe0d39a7000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:54:52.419 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:52.314+0000 7fe0d39a7000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:54:52.419 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:52.418+0000 7fe0d39a7000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:54:52.743 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:52.498+0000 7fe0d39a7000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:54:52.743 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:52.659+0000 7fe0d39a7000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:54:52.743 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:52.742+0000 7fe0d39a7000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:54:54.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:53.780+0000 7fe0d39a7000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:54:54.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:53.909+0000 7fe0d39a7000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:54:54.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:54.110+0000 7fe0d39a7000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:54:54.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:54.198+0000 7fe0d39a7000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:54:54.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:54.332+0000 7fe0d39a7000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:54:54.956 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:54.654+0000 7fe0d39a7000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:54:54.956 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:54.873+0000 7fe0d39a7000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:54:54.956 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:54.956+0000 7fe0d39a7000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:54:55.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:55 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:55.124+0000 7fe0d39a7000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:54:56.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:54:56.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2024-02-10T05:54:56.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsid": "ce44e742-c7d8-11ee-95b8-87774f69a715", 2024-02-10T05:54:56.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "health": { 2024-02-10T05:54:56.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T05:54:56.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T05:54:56.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T05:54:56.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:56.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T05:54:56.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T05:54:56.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 0 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "a" 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T05:54:56.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T05:54:56.062 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "available": false, 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "iostat", 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "nfs", 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "restful" 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T05:54:56.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:56.064 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modified": "2024-02-10T05:54:37.242795+0000", 2024-02-10T05:54:56.064 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:54:56.064 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:56.064 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T05:54:56.064 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2024-02-10T05:54:56.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:56 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3043575654' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:54:56.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:56 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:56.055+0000 7fe0d39a7000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:54:56.638 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:56 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:56.361+0000 7fe0d39a7000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:54:56.639 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:56 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:56.454+0000 7fe0d39a7000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:54:56.639 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:54:56 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:54:56.604+0000 7fe0d39a7000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:54:56.646 INFO:teuthology.orchestra.run.smithi139.stderr:mgr not available, waiting (2/15)... 2024-02-10T05:54:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: Activating manager daemon y 2024-02-10T05:54:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: mgrmap e2: y(active, starting, since 0.00285788s) 2024-02-10T05:54:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: from='mgr.14100 172.21.15.139:0/1822207486' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:54:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: from='mgr.14100 172.21.15.139:0/1822207486' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:54:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: from='mgr.14100 172.21.15.139:0/1822207486' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:54:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: from='mgr.14100 172.21.15.139:0/1822207486' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:54:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: from='mgr.14100 172.21.15.139:0/1822207486' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:54:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: Manager daemon y is now available 2024-02-10T05:54:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: from='mgr.14100 172.21.15.139:0/1822207486' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:54:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: from='mgr.14100 172.21.15.139:0/1822207486' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:54:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: from='mgr.14100 172.21.15.139:0/1822207486' entity='mgr.y' 2024-02-10T05:54:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: from='mgr.14100 172.21.15.139:0/1822207486' entity='mgr.y' 2024-02-10T05:54:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:57 smithi139 ceph-mon[24727]: from='mgr.14100 172.21.15.139:0/1822207486' entity='mgr.y' 2024-02-10T05:54:58.873 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:58 smithi139 ceph-mon[24727]: mgrmap e3: y(active, since 1.0062s) 2024-02-10T05:54:59.337 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsid": "ce44e742-c7d8-11ee-95b8-87774f69a715", 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "health": { 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 0 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T05:54:59.338 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "a" 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T05:54:59.339 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:59.343 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "available": true, 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "iostat", 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "nfs", 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "restful" 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T05:54:59.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:54:59.345 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modified": "2024-02-10T05:54:37.242795+0000", 2024-02-10T05:54:59.345 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:54:59.345 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2024-02-10T05:54:59.345 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T05:54:59.345 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2024-02-10T05:54:59.893 INFO:teuthology.orchestra.run.smithi139.stderr:mgr is available 2024-02-10T05:55:00.007 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:59 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3139226310' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:55:00.007 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:54:59 smithi139 ceph-mon[24727]: mgrmap e4: y(active, since 3s) 2024-02-10T05:55:00.474 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: [global] 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: fsid = ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: [mgr] 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: [osd] 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T05:55:00.475 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T05:55:01.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:00 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1780007804' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T05:55:01.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:00 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1780007804' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T05:55:02.151 INFO:teuthology.orchestra.run.smithi139.stderr:Enabling cephadm module... 2024-02-10T05:55:03.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:02 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1984861064' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T05:55:03.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:02 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ignoring --setuser ceph since I am not root 2024-02-10T05:55:03.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:02 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ignoring --setgroup ceph since I am not root 2024-02-10T05:55:03.365 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:03.274+0000 7f2a438ef000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:55:03.365 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:03.364+0000 7f2a438ef000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:55:03.972 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:03 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1984861064' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T05:55:03.972 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:03 smithi139 ceph-mon[24727]: mgrmap e5: y(active, since 6s) 2024-02-10T05:55:04.053 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2024-02-10T05:55:04.053 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-10T05:55:04.053 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "available": true, 2024-02-10T05:55:04.053 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-10T05:55:04.054 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T05:55:04.054 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2024-02-10T05:55:04.302 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:03.972+0000 7f2a438ef000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:55:04.303 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:04 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:04.062+0000 7f2a438ef000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:55:04.303 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:04 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:04.149+0000 7f2a438ef000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:55:04.303 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:04 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:04.302+0000 7f2a438ef000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:55:04.576 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:04 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:04.384+0000 7f2a438ef000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:55:04.577 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:04 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:04.491+0000 7f2a438ef000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:55:04.577 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:04 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:04.576+0000 7f2a438ef000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:55:04.638 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for the mgr to restart... 2024-02-10T05:55:04.638 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mgr epoch 5... 2024-02-10T05:55:04.832 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:04 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/36176789' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T05:55:04.832 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:04 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:04.744+0000 7f2a438ef000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:55:04.833 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:04 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:04.832+0000 7f2a438ef000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:55:06.160 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:05 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:05.850+0000 7f2a438ef000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:55:06.160 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:05 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:05.976+0000 7f2a438ef000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:55:06.160 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:06 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:06.160+0000 7f2a438ef000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:55:06.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:06 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:06.244+0000 7f2a438ef000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:55:06.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:06 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:06.375+0000 7f2a438ef000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:55:06.984 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:06 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:06.686+0000 7f2a438ef000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:55:06.984 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:06 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:06.902+0000 7f2a438ef000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:55:06.984 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:06 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:06.982+0000 7f2a438ef000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:55:07.254 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:07 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:07.146+0000 7f2a438ef000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:55:08.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:08.040+0000 7f2a438ef000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:55:08.644 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:08.337+0000 7f2a438ef000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:55:08.644 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:08.429+0000 7f2a438ef000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:55:08.644 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:08.576+0000 7f2a438ef000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:55:09.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:08 smithi139 ceph-mon[24727]: Active manager daemon y restarted 2024-02-10T05:55:09.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:08 smithi139 ceph-mon[24727]: Activating manager daemon y 2024-02-10T05:55:09.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:08 smithi139 ceph-mon[24727]: osdmap e2: 0 total, 0 up, 0 in 2024-02-10T05:55:09.656 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2024-02-10T05:55:09.656 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-10T05:55:09.656 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T05:55:09.656 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2024-02-10T05:55:09.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:55:09] ENGINE Bus STARTING 2024-02-10T05:55:09.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:55:09] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T05:55:09.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:55:09] ENGINE Bus STARTED 2024-02-10T05:55:09.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: mgrmap e6: y(active, starting, since 0.0527986s) 2024-02-10T05:55:09.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: Manager daemon y is now available 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:55:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:09 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:10.219 INFO:teuthology.orchestra.run.smithi139.stderr:mgr epoch 5 is available 2024-02-10T05:55:10.219 INFO:teuthology.orchestra.run.smithi139.stderr:Setting orchestrator backend to cephadm... 2024-02-10T05:55:10.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:10 smithi139 ceph-mon[24727]: [10/Feb/2024:05:55:09] ENGINE Bus STARTING 2024-02-10T05:55:10.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:10 smithi139 ceph-mon[24727]: [10/Feb/2024:05:55:09] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T05:55:10.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:10 smithi139 ceph-mon[24727]: [10/Feb/2024:05:55:09] ENGINE Bus STARTED 2024-02-10T05:55:10.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:10 smithi139 ceph-mon[24727]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T05:55:10.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:10 smithi139 ceph-mon[24727]: mgrmap e7: y(active, since 1.05813s) 2024-02-10T05:55:10.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:10 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:10.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:10 smithi139 ceph-mon[24727]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T05:55:11.740 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:11 smithi139 ceph-mon[24727]: mgrmap e8: y(active, since 2s) 2024-02-10T05:55:11.740 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:11 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:11.740 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:11 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:12.020 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: value unchanged 2024-02-10T05:55:12.604 INFO:teuthology.orchestra.run.smithi139.stderr:Generating ssh key... 2024-02-10T05:55:12.921 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:12 smithi139 ceph-mon[24727]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:13.654 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: Generating public/private rsa key pair. 2024-02-10T05:55:13.654 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: Your identification has been saved in /tmp/tmp1g87ovsl/key. 2024-02-10T05:55:13.654 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: Your public key has been saved in /tmp/tmp1g87ovsl/key.pub. 2024-02-10T05:55:13.654 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: The key fingerprint is: 2024-02-10T05:55:13.654 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: SHA256:laL4MDCCq/AJEkOL42eBmWeJ3hcjC9YqKQVxadwCMdI ceph-ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: The key's randomart image is: 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: +---[RSA 3072]----+ 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: |BB.o | 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: |=+E o . | 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: |BX+O o . o | 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: |+O*o+.o. o | 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: |O+ =+.. S | 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: |*oo..+ | 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: |. o . | 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: | | 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: | | 2024-02-10T05:55:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: +----[SHA256]-----+ 2024-02-10T05:55:14.025 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-mon[24727]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:14.025 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:14.025 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:13 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:14.500 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmdDs22wMD4k6G6FtDK8FhkfVQyFw0vkEkoHI9kcj1IktqW2r05OOhGkw4BCI7JHZFA8ssKGDp+hFSoutbSP3GnfTzSzvqbXhJ9Rd0LK7HG9OhiRNwIn27TAGg4+pE4j/636E7RPcvc2f+qx23MBdqUGYbX3RiZpaVTI7ZDOQnNJK2d58u0Edmglw2b7qb8RHXrYtg4HauU9JyDZqBYi6XqtrcMk5UKpRUakgMOnYb+F6aQQcXKZ1jRHN4fa1QpRijvqo2vFE/LV8Sb9MtXuPz+U1DmFQglmPNVxiHtn3gHspDVgI6CyAFT5Q55X0VTkYtLNCSXbWibBXIRdFDudmGrMsbFIFyj5KFrx6GN4Mo63pT0xWS69QqWp4uzTWn2MPuJpY2j/nR/WCLJD/NDNRzUcEHQPCv/DDWX1/Zkf3wALhJ7yBHouOTrM/0iKZPTSaIzndIEe1K20hUp3vzrWGbjBkPKx+tZd+X5kB9yxRcFK8olAY2/X8dI0+2AiHjnyk= ceph-ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:55:15.028 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:14 smithi139 ceph-mon[24727]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:15.028 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:14 smithi139 ceph-mon[24727]: Generating ssh key... 2024-02-10T05:55:15.102 INFO:teuthology.orchestra.run.smithi139.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:55:15.102 INFO:teuthology.orchestra.run.smithi139.stderr:Adding key to root@localhost authorized_keys... 2024-02-10T05:55:15.102 INFO:teuthology.orchestra.run.smithi139.stderr:Adding host smithi139... 2024-02-10T05:55:15.972 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:15 smithi139 ceph-mon[24727]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:16.654 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Added host 'smithi139' with addr '172.21.15.139' 2024-02-10T05:55:16.953 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:16 smithi139 ceph-mon[24727]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi139", "addr": "172.21.15.139", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:16.953 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:16 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:16.953 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:16 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:17.239 INFO:teuthology.orchestra.run.smithi139.stderr:Deploying unmanaged mon service... 2024-02-10T05:55:17.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:17 smithi139 ceph-mon[24727]: Deploying cephadm binary to smithi139 2024-02-10T05:55:17.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:17 smithi139 ceph-mon[24727]: Added host smithi139 2024-02-10T05:55:17.839 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-10T05:55:18.647 INFO:teuthology.orchestra.run.smithi139.stderr:Deploying unmanaged mgr service... 2024-02-10T05:55:19.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:18 smithi139 ceph-mon[24727]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:19.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:18 smithi139 ceph-mon[24727]: Saving service mon spec with placement count:5 2024-02-10T05:55:19.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:18 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:19.316 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-10T05:55:20.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:20 smithi139 ceph-mon[24727]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:20.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:20 smithi139 ceph-mon[24727]: Saving service mgr spec with placement count:2 2024-02-10T05:55:20.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:20 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:20.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:20 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:22.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:21 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:22.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:21 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/82905168' entity='client.admin' 2024-02-10T05:55:23.421 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:23 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:23.422 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:23 smithi139 ceph-mon[24727]: from='mgr.14118 172.21.15.139:0/2265300490' entity='mgr.y' 2024-02-10T05:55:23.699 INFO:teuthology.orchestra.run.smithi139.stderr:Enabling the dashboard module... 2024-02-10T05:55:24.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:24 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/815941848' entity='client.admin' 2024-02-10T05:55:25.184 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:25 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/188734815' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T05:55:25.449 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:25 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ignoring --setuser ceph since I am not root 2024-02-10T05:55:25.449 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:25 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ignoring --setgroup ceph since I am not root 2024-02-10T05:55:25.726 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:25 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:25.635+0000 7f2fcf35a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:55:25.726 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:25 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:25.725+0000 7f2fcf35a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:55:26.333 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:26 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/188734815' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T05:55:26.333 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:26 smithi139 ceph-mon[24727]: mgrmap e9: y(active, since 16s) 2024-02-10T05:55:26.527 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2024-02-10T05:55:26.527 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-10T05:55:26.527 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "available": true, 2024-02-10T05:55:26.527 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-10T05:55:26.527 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T05:55:26.527 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2024-02-10T05:55:26.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:26 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:26.332+0000 7f2fcf35a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:55:26.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:26 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:26.423+0000 7f2fcf35a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:55:26.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:26 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:26.508+0000 7f2fcf35a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:55:26.936 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:26 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:26.663+0000 7f2fcf35a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:55:26.936 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:26 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:26.743+0000 7f2fcf35a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:55:26.936 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:26 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:26.852+0000 7f2fcf35a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:55:26.936 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:26 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:26.935+0000 7f2fcf35a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:55:27.176 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for the mgr to restart... 2024-02-10T05:55:27.176 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mgr epoch 9... 2024-02-10T05:55:27.298 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:27 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2724893335' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T05:55:27.298 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:27 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:27.097+0000 7f2fcf35a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:55:27.298 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:27 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:27.185+0000 7f2fcf35a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:55:28.525 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:28.207+0000 7f2fcf35a000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:55:28.526 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:28.334+0000 7f2fcf35a000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:55:28.526 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:28.525+0000 7f2fcf35a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:55:28.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:28.614+0000 7f2fcf35a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:55:28.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:28.755+0000 7f2fcf35a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:55:29.312 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:29.086+0000 7f2fcf35a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:55:29.573 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:29.311+0000 7f2fcf35a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:55:29.573 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:29.398+0000 7f2fcf35a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:55:29.573 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:29.572+0000 7f2fcf35a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:55:30.789 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:30 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:30.498+0000 7f2fcf35a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:55:30.877 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:30 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:30.788+0000 7f2fcf35a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:55:31.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:30 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:30.884+0000 7f2fcf35a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:55:31.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:31 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:55:31.034+0000 7f2fcf35a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:55:32.053 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:31 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:55:31] ENGINE Bus STARTING 2024-02-10T05:55:32.053 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:31 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:55:31] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T05:55:32.053 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:55:31 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:55:31] ENGINE Bus STARTED 2024-02-10T05:55:32.114 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2024-02-10T05:55:32.114 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-10T05:55:32.114 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T05:55:32.114 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2024-02-10T05:55:32.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: Active manager daemon y restarted 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: Activating manager daemon y 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: osdmap e3: 0 total, 0 up, 0 in 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: mgrmap e10: y(active, starting, since 0.0552485s) 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: Manager daemon y is now available 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:55:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:32.885 INFO:teuthology.orchestra.run.smithi139.stderr:mgr epoch 9 is available 2024-02-10T05:55:32.886 INFO:teuthology.orchestra.run.smithi139.stderr:Generating a dashboard self-signed certificate... 2024-02-10T05:55:33.179 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:33 smithi139 ceph-mon[24727]: [10/Feb/2024:05:55:31] ENGINE Bus STARTING 2024-02-10T05:55:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:33 smithi139 ceph-mon[24727]: [10/Feb/2024:05:55:31] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T05:55:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:33 smithi139 ceph-mon[24727]: [10/Feb/2024:05:55:31] ENGINE Bus STARTED 2024-02-10T05:55:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:33 smithi139 ceph-mon[24727]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T05:55:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:33 smithi139 ceph-mon[24727]: mgrmap e11: y(active, since 1.058s) 2024-02-10T05:55:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:33 smithi139 ceph-mon[24727]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T05:55:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:33 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:33.679 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-10T05:55:34.375 INFO:teuthology.orchestra.run.smithi139.stderr:Creating initial admin user... 2024-02-10T05:55:34.751 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:34 smithi139 ceph-mon[24727]: mgrmap e12: y(active, since 2s) 2024-02-10T05:55:34.751 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:34 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:34.751 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:34 smithi139 ceph-mon[24727]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:34.751 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:34 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:34.751 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:34 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:36.393 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$BFKsbqVSec3Y17jwCKdYPegQjDlVDf3Fk6gSt896E9Ra9tQMIlwE6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707544536, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T05:55:36.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:36 smithi139 ceph-mon[24727]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:37.138 INFO:teuthology.orchestra.run.smithi139.stderr:Fetching dashboard port number... 2024-02-10T05:55:37.784 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:37 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:37.785 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 8443 2024-02-10T05:55:38.387 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: disabled 2024-02-10T05:55:38.395 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2024-02-10T05:55:38.395 INFO:teuthology.orchestra.run.smithi139.stderr:firewalld.service is not enabled 2024-02-10T05:55:38.396 INFO:teuthology.orchestra.run.smithi139.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T05:55:38.398 INFO:teuthology.orchestra.run.smithi139.stderr:Ceph Dashboard is now available at: 2024-02-10T05:55:38.398 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-02-10T05:55:38.398 INFO:teuthology.orchestra.run.smithi139.stderr: URL: https://smithi139.front.sepia.ceph.com:8443/ 2024-02-10T05:55:38.398 INFO:teuthology.orchestra.run.smithi139.stderr: User: admin 2024-02-10T05:55:38.398 INFO:teuthology.orchestra.run.smithi139.stderr: Password: yzt6iagm28 2024-02-10T05:55:38.398 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-02-10T05:55:38.398 INFO:teuthology.orchestra.run.smithi139.stderr:Enabling autotune for osd_memory_target 2024-02-10T05:55:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:38 smithi139 ceph-mon[24727]: mgrmap e13: y(active, since 6s) 2024-02-10T05:55:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:38 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1043186499' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T05:55:40.305 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-10T05:55:40.932 INFO:teuthology.orchestra.run.smithi139.stderr:You can access the Ceph CLI with: 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr: ceph telemetry on 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr:For more information see: 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-02-10T05:55:40.933 INFO:teuthology.orchestra.run.smithi139.stderr:Bootstrap complete. 2024-02-10T05:55:40.971 INFO:tasks.cephadm:Fetching config... 2024-02-10T05:55:40.971 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-02-10T05:55:40.971 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T05:55:40.986 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T05:55:40.987 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-02-10T05:55:40.987 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T05:55:41.042 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T05:55:41.043 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-02-10T05:55:41.043 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-10T05:55:41.110 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T05:55:41.111 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-02-10T05:55:41.111 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T05:55:41.166 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T05:55:41.166 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmdDs22wMD4k6G6FtDK8FhkfVQyFw0vkEkoHI9kcj1IktqW2r05OOhGkw4BCI7JHZFA8ssKGDp+hFSoutbSP3GnfTzSzvqbXhJ9Rd0LK7HG9OhiRNwIn27TAGg4+pE4j/636E7RPcvc2f+qx23MBdqUGYbX3RiZpaVTI7ZDOQnNJK2d58u0Edmglw2b7qb8RHXrYtg4HauU9JyDZqBYi6XqtrcMk5UKpRUakgMOnYb+F6aQQcXKZ1jRHN4fa1QpRijvqo2vFE/LV8Sb9MtXuPz+U1DmFQglmPNVxiHtn3gHspDVgI6CyAFT5Q55X0VTkYtLNCSXbWibBXIRdFDudmGrMsbFIFyj5KFrx6GN4Mo63pT0xWS69QqWp4uzTWn2MPuJpY2j/nR/WCLJD/NDNRzUcEHQPCv/DDWX1/Zkf3wALhJ7yBHouOTrM/0iKZPTSaIzndIEe1K20hUp3vzrWGbjBkPKx+tZd+X5kB9yxRcFK8olAY2/X8dI0+2AiHjnyk= ceph-ce44e742-c7d8-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T05:55:41.257 INFO:teuthology.orchestra.run.smithi139.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmdDs22wMD4k6G6FtDK8FhkfVQyFw0vkEkoHI9kcj1IktqW2r05OOhGkw4BCI7JHZFA8ssKGDp+hFSoutbSP3GnfTzSzvqbXhJ9Rd0LK7HG9OhiRNwIn27TAGg4+pE4j/636E7RPcvc2f+qx23MBdqUGYbX3RiZpaVTI7ZDOQnNJK2d58u0Edmglw2b7qb8RHXrYtg4HauU9JyDZqBYi6XqtrcMk5UKpRUakgMOnYb+F6aQQcXKZ1jRHN4fa1QpRijvqo2vFE/LV8Sb9MtXuPz+U1DmFQglmPNVxiHtn3gHspDVgI6CyAFT5Q55X0VTkYtLNCSXbWibBXIRdFDudmGrMsbFIFyj5KFrx6GN4Mo63pT0xWS69QqWp4uzTWn2MPuJpY2j/nR/WCLJD/NDNRzUcEHQPCv/DDWX1/Zkf3wALhJ7yBHouOTrM/0iKZPTSaIzndIEe1K20hUp3vzrWGbjBkPKx+tZd+X5kB9yxRcFK8olAY2/X8dI0+2AiHjnyk= ceph-ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:55:41.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmdDs22wMD4k6G6FtDK8FhkfVQyFw0vkEkoHI9kcj1IktqW2r05OOhGkw4BCI7JHZFA8ssKGDp+hFSoutbSP3GnfTzSzvqbXhJ9Rd0LK7HG9OhiRNwIn27TAGg4+pE4j/636E7RPcvc2f+qx23MBdqUGYbX3RiZpaVTI7ZDOQnNJK2d58u0Edmglw2b7qb8RHXrYtg4HauU9JyDZqBYi6XqtrcMk5UKpRUakgMOnYb+F6aQQcXKZ1jRHN4fa1QpRijvqo2vFE/LV8Sb9MtXuPz+U1DmFQglmPNVxiHtn3gHspDVgI6CyAFT5Q55X0VTkYtLNCSXbWibBXIRdFDudmGrMsbFIFyj5KFrx6GN4Mo63pT0xWS69QqWp4uzTWn2MPuJpY2j/nR/WCLJD/NDNRzUcEHQPCv/DDWX1/Zkf3wALhJ7yBHouOTrM/0iKZPTSaIzndIEe1K20hUp3vzrWGbjBkPKx+tZd+X5kB9yxRcFK8olAY2/X8dI0+2AiHjnyk= ceph-ce44e742-c7d8-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T05:55:41.318 INFO:teuthology.orchestra.run.smithi150.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmdDs22wMD4k6G6FtDK8FhkfVQyFw0vkEkoHI9kcj1IktqW2r05OOhGkw4BCI7JHZFA8ssKGDp+hFSoutbSP3GnfTzSzvqbXhJ9Rd0LK7HG9OhiRNwIn27TAGg4+pE4j/636E7RPcvc2f+qx23MBdqUGYbX3RiZpaVTI7ZDOQnNJK2d58u0Edmglw2b7qb8RHXrYtg4HauU9JyDZqBYi6XqtrcMk5UKpRUakgMOnYb+F6aQQcXKZ1jRHN4fa1QpRijvqo2vFE/LV8Sb9MtXuPz+U1DmFQglmPNVxiHtn3gHspDVgI6CyAFT5Q55X0VTkYtLNCSXbWibBXIRdFDudmGrMsbFIFyj5KFrx6GN4Mo63pT0xWS69QqWp4uzTWn2MPuJpY2j/nR/WCLJD/NDNRzUcEHQPCv/DDWX1/Zkf3wALhJ7yBHouOTrM/0iKZPTSaIzndIEe1K20hUp3vzrWGbjBkPKx+tZd+X5kB9yxRcFK8olAY2/X8dI0+2AiHjnyk= ceph-ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:55:41.333 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T05:55:41.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:41 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1614608373' entity='client.admin' 2024-02-10T05:55:44.038 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:44.039 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:44.039 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:44.039 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:44.039 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:44.583 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-10T05:55:44.583 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-10T05:55:45.040 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:44 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2329473104' entity='client.admin' 2024-02-10T05:55:45.040 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:44 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:45.040 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:44 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:47.459 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi150 2024-02-10T05:55:47.460 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-10T05:55:47.460 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.conf 2024-02-10T05:55:47.478 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-10T05:55:47.478 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:55:47.535 INFO:tasks.cephadm:Adding host smithi150 to orchestrator... 2024-02-10T05:55:47.535 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch host add smithi150 2024-02-10T05:55:47.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:47 smithi139 ceph-mon[24727]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:47.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:47.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:47.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:47.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:47.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:47 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T05:55:47.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:47 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:55:47.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:47.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:50.372 INFO:teuthology.orchestra.run.smithi139.stdout:Added host 'smithi150' with addr '172.21.15.150' 2024-02-10T05:55:50.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:50 smithi139 ceph-mon[24727]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:50.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:50 smithi139 ceph-mon[24727]: Deploying cephadm binary to smithi150 2024-02-10T05:55:50.998 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T05:55:51.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:51 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:51.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:51 smithi139 ceph-mon[24727]: Added host smithi150 2024-02-10T05:55:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:51 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:51 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:51 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:51 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:51 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:51 smithi139 ceph-mon[24727]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:55:53.001 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T05:55:53.002 INFO:teuthology.orchestra.run.smithi139.stdout:[{"addr": "172.21.15.139", "hostname": "smithi139", "labels": [], "status": ""}, {"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}] 2024-02-10T05:55:53.232 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:52 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:53.633 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T05:55:53.633 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T05:55:54.221 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:53 smithi139 ceph-mon[24727]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:55:54.221 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:53 smithi139 ceph-mon[24727]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:55:55.781 INFO:teuthology.orchestra.run.smithi139.stderr:adjusted tunables profile to default 2024-02-10T05:55:56.079 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:55 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:56.079 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:55 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:56.079 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:55 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:56.079 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:55 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:56.079 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:55 smithi139 ceph-mon[24727]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:55:56.079 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:55 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3235731768' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:55:56.365 INFO:tasks.cephadm:Adding mon.a on smithi139 2024-02-10T05:55:56.398 INFO:tasks.cephadm:Adding mon.c on smithi139 2024-02-10T05:55:56.398 INFO:tasks.cephadm:Adding mon.b on smithi150 2024-02-10T05:55:56.398 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply mon '3;smithi139:172.21.15.139=a;smithi139:[v2:172.21.15.139:3301,v1:172.21.15.139:6790]=c;smithi150:172.21.15.150=b' 2024-02-10T05:55:57.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:56 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3235731768' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:55:57.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:56 smithi139 ceph-mon[24727]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:55:58.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:57 smithi139 ceph-mon[24727]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:55:58.421 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled mon update... 2024-02-10T05:55:59.097 DEBUG:teuthology.orchestra.run.smithi139:mon.c> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.c.service 2024-02-10T05:55:59.100 DEBUG:teuthology.orchestra.run.smithi150:mon.b> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.b.service 2024-02-10T05:55:59.102 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:55:59.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:55:59.741 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi139:172.21.15.139=a;smithi139:[v2:172.21.15.139:3301,v1:172.21.15.139:6790]=c;smithi150:172.21.15.150=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:59.741 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: Saving service mon spec with placement smithi139:172.21.15.139=a;smithi139:[v2:172.21.15.139:3301,v1:172.21.15.139:6790]=c;smithi150:172.21.15.150=b;count:3 2024-02-10T05:55:59.741 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:59.741 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:59.741 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:59.741 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:59.741 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T05:55:59.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:55:59.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:59.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:55:59.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:55:59.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:59.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: Deploying daemon mon.c on smithi139 2024-02-10T05:55:59.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:55:59 smithi139 ceph-mon[24727]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:01.106 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-10T05:56:01.106 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"ce44e742-c7d8-11ee-95b8-87774f69a715","modified":"2024-02-10T05:54:34.334911Z","created":"2024-02-10T05:54:34.334911Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:56:01.108 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-02-10T05:56:01.512 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:01 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/2901703153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:56:01.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:01 smithi139 systemd[1]: Starting Ceph mon.c for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:56:02.217 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:01 smithi139 podman[29435]: 2024-02-10 05:56:01.934976522 +0000 UTC m=+0.108436920 container create 0ed77770af21057814ee76b70dc9ede38a774e7c854b012d6e878399c56e048d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, version=8, distribution-scope=public, RELEASE=HEAD, vcs-type=git, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, ceph=True, architecture=x86_64) 2024-02-10T05:56:02.217 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:01 smithi139 podman[29435]: 2024-02-10 05:56:01.845931982 +0000 UTC m=+0.019392383 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-02-10T05:56:02.217 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 podman[29435]: 2024-02-10 05:56:02.112165746 +0000 UTC m=+0.285626139 container init 0ed77770af21057814ee76b70dc9ede38a774e7c854b012d6e878399c56e048d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c, ceph=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.openshift.expose-services=) 2024-02-10T05:56:02.217 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 podman[29435]: 2024-02-10 05:56:02.115032063 +0000 UTC m=+0.288492459 container start 0ed77770af21057814ee76b70dc9ede38a774e7c854b012d6e878399c56e048d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_CLEAN=True, distribution-scope=public, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.buildah.version=1.19.8, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, com.redhat.component=centos-stream-container, io.openshift.expose-services=) 2024-02-10T05:56:02.217 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: pidfile_write: ignore empty --pid-file 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: load: jerasure load: lrc 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: RocksDB version: 6.15.5 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Compile date Apr 18 2022 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: DB SUMMARY 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: DB Session ID: AZ4V66WVANYIE8DR26FY 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-02-10T05:56:02.218 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:56:02.219 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:56:02.219 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:56:02.219 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:56:02.219 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.env: 0x5629492c6860 2024-02-10T05:56:02.219 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.fs: Posix File System 2024-02-10T05:56:02.219 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.info_log: 0x56294a761dc0 2024-02-10T05:56:02.219 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:56:02.219 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.statistics: (nil) 2024-02-10T05:56:02.219 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.use_fsync: 0 2024-02-10T05:56:02.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:56:02.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:56:02.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:56:02.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:56:02.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:56:02.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:56:02.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.db_log_dir: 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:56:02.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.write_buffer_manager: 0x56294a852270 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.unordered_write: 0 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:56:02.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.row_cache: None 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.wal_filter: None 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.preserve_deletes: 0 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:56:02.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_open_files: -1 2024-02-10T05:56:02.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Compression algorithms supported: 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: kZSTD supported: 0 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:56:02.226 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.merge_operator: 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_filter: None 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56294a72dd00) 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_type: 0 2024-02-10T05:56:02.227 INFO:journalctl@ceph.mon.c.smithi139.stdout: data_block_index_type: 0 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_shortening: 1 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: hash_index_allow_collision: 1 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: checksum: 1 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: no_block_cache: 0 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache: 0x56294a798170 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_options: 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: capacity : 536870912 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: num_shard_bits : 4 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: strict_capacity_limit : 0 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_compressed: (nil) 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: persistent_cache: (nil) 2024-02-10T05:56:02.228 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_size: 4096 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_size_deviation: 10 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_restart_interval: 16 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_block_restart_interval: 1 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: metadata_block_size: 4096 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: partition_filters: 0 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: use_delta_encoding: 1 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: whole_key_filtering: 1 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: verify_compression: 0 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: format_version: 4 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: enable_index_compression: 1 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_align: 0 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:56:02.229 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compression: NoCompression 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.num_levels: 7 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:56:02.230 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:56:02.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:56:02.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:56:02.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:56:02.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:56:02.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:56:02.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:56:02.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:56:02.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:56:02.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T05:56:02.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:56:02.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:56:02.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.ttl: 2592000 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:56:02.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:56:02.235 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:56:02.235 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:56:02.235 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:56:02.235 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:56:02.235 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:56:02.235 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-02-10T05:56:02.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-02-10T05:56:02.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544562140577, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T05:56:02.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-02-10T05:56:02.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T05:56:02.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544562140866, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707544562, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "03962755-a1db-4d2d-b0c1-0918fe82ed66", "db_session_id": "AZ4V66WVANYIE8DR26FY"}} 2024-02-10T05:56:02.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-02-10T05:56:02.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544562141193, "job": 1, "event": "recovery_finished"} 2024-02-10T05:56:02.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:56:02.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x56294a77ea80 2024-02-10T05:56:02.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: DB pointer 0x56294a78e000 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** DB Stats ** 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:56:02.237 INFO:journalctl@ceph.mon.c.smithi139.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:56:02.238 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T05:56:02.239 INFO:journalctl@ceph.mon.c.smithi139.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-10T05:56:02.240 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: using public_addrv [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: starting mon.c rank -1 at public addrs [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] at bind addrs [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(???) e0 preinit fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).mds e1 new map 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout: e1 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout: legacy client fscid: -1 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout: No filesystems configured 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T05:56:02.241 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi139:172.21.15.139=a;smithi139:[v2:172.21.15.139:3301,v1:172.21.15.139:6790]=c;smithi150:172.21.15.150=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: Saving service mon spec with placement smithi139:172.21.15.139=a;smithi139:[v2:172.21.15.139:3301,v1:172.21.15.139:6790]=c;smithi150:172.21.15.150=b;count:3 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:02.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:02.243 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:02.243 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: Deploying daemon mon.c on smithi139 2024-02-10T05:56:02.243 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:02.243 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/2901703153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:56:02.243 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-10T05:56:02.243 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 ceph-mon[29453]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-10T05:56:02.243 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 bash[29435]: 0ed77770af21057814ee76b70dc9ede38a774e7c854b012d6e878399c56e048d 2024-02-10T05:56:02.243 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:02 smithi139 systemd[1]: Started Ceph mon.c for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:56:02.644 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:56:02.645 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:56:07.517 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-10T05:56:07.517 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":2,"fsid":"ce44e742-c7d8-11ee-95b8-87774f69a715","modified":"2024-02-10T05:56:02.158256Z","created":"2024-02-10T05:54:34.334911Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3301","nonce":0},{"type":"v1","addr":"172.21.15.139:6790","nonce":0}]},"addr":"172.21.15.139:6790/0","public_addr":"172.21.15.139:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T05:56:07.519 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 2 2024-02-10T05:56:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:56:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: mon.a calling monitor election 2024-02-10T05:56:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: mon.c calling monitor election 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: monmap e2: 2 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: fsmap 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: mgrmap e13: y(active, since 36s) 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: overall HEALTH_OK 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:07.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:07.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:56:07.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: mon.a calling monitor election 2024-02-10T05:56:07.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:07.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: mon.c calling monitor election 2024-02-10T05:56:07.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:07.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: monmap e2: 2 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: fsmap 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: mgrmap e13: y(active, since 36s) 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: overall HEALTH_OK 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:07.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:08 smithi139 ceph-mon[24727]: Deploying daemon mon.b on smithi150 2024-02-10T05:56:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:08 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/220634138' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:56:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:08 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:08 smithi139 ceph-mon[29453]: Deploying daemon mon.b on smithi150 2024-02-10T05:56:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:08 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/220634138' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:56:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:08 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:09 smithi139 ceph-mon[24727]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:10.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:09 smithi139 ceph-mon[29453]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:10.788 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:56:10.788 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:56:11.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:11 smithi150 systemd[1]: Starting Ceph mon.b for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:56:12.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 podman[25520]: 2024-02-10 05:56:12.119589416 +0000 UTC m=+0.211983275 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-02-10T05:56:12.582 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-10T05:56:12.583 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":2,"fsid":"ce44e742-c7d8-11ee-95b8-87774f69a715","modified":"2024-02-10T05:56:02.158256Z","created":"2024-02-10T05:54:34.334911Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3301","nonce":0},{"type":"v1","addr":"172.21.15.139:6790","nonce":0}]},"addr":"172.21.15.139:6790/0","public_addr":"172.21.15.139:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T05:56:12.584 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 2 2024-02-10T05:56:12.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:12 smithi139 ceph-mon[24727]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:12.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:12 smithi139 ceph-mon[29453]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:12.770 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 podman[25520]: 2024-02-10 05:56:12.486406264 +0000 UTC m=+0.578800120 container create 51b65c6e04ce76ace48f94651a265b2a5ac31b0c55474b3633996e1cfe795eed (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, maintainer=Guillaume Abrioux , io.openshift.expose-services=, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, ceph=True, version=8, name=centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, vcs-type=git, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T05:56:12.770 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 podman[25520]: 2024-02-10 05:56:12.655637186 +0000 UTC m=+0.748031044 container init 51b65c6e04ce76ace48f94651a265b2a5ac31b0c55474b3633996e1cfe795eed (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, release=754, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, distribution-scope=public, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-10T05:56:12.770 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 podman[25520]: 2024-02-10 05:56:12.658751532 +0000 UTC m=+0.751145398 container start 51b65c6e04ce76ace48f94651a265b2a5ac31b0c55474b3633996e1cfe795eed (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, build-date=2022-05-03T08:36:31.336870, version=8, architecture=x86_64, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, name=centos-stream, vcs-type=git, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=) 2024-02-10T05:56:12.770 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:56:12.770 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-10T05:56:12.770 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: pidfile_write: ignore empty --pid-file 2024-02-10T05:56:12.770 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: load: jerasure load: lrc 2024-02-10T05:56:12.770 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: RocksDB version: 6.15.5 2024-02-10T05:56:12.770 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T05:56:12.770 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Compile date Apr 18 2022 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: DB SUMMARY 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: DB Session ID: 2ZPIML5CKILTJC5YRW01 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.env: 0x56174a6c6860 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.fs: Posix File System 2024-02-10T05:56:12.771 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.info_log: 0x56174ca45dc0 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.statistics: (nil) 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.use_fsync: 0 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:56:12.772 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.db_log_dir: 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.write_buffer_manager: 0x56174cb36240 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T05:56:12.773 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.unordered_write: 0 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.row_cache: None 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.wal_filter: None 2024-02-10T05:56:12.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:56:12.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:56:12.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.preserve_deletes: 0 2024-02-10T05:56:12.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:56:12.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:56:12.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_open_files: -1 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:56:12.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Compression algorithms supported: 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: kZSTD supported: 0 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.merge_operator: 2024-02-10T05:56:12.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_filter: None 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56174ca11d00) 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_type: 0 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout: data_block_index_type: 0 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_shortening: 1 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout: hash_index_allow_collision: 1 2024-02-10T05:56:12.780 INFO:journalctl@ceph.mon.b.smithi150.stdout: checksum: 1 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: no_block_cache: 0 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache: 0x56174ca7c170 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_options: 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: capacity : 536870912 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: num_shard_bits : 4 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: strict_capacity_limit : 0 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_compressed: (nil) 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: persistent_cache: (nil) 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_size: 4096 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_size_deviation: 10 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_restart_interval: 16 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_block_restart_interval: 1 2024-02-10T05:56:12.781 INFO:journalctl@ceph.mon.b.smithi150.stdout: metadata_block_size: 4096 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout: partition_filters: 0 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout: use_delta_encoding: 1 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout: whole_key_filtering: 1 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout: verify_compression: 0 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout: format_version: 4 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout: enable_index_compression: 1 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_align: 0 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compression: NoCompression 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:56:12.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.num_levels: 7 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:56:12.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:56:12.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:56:12.785 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:56:12.786 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.ttl: 2592000 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544572685053, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T05:56:12.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544572685351, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707544572, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "8e00fb6d-d07f-4008-a352-3511f38022a2", "db_session_id": "2ZPIML5CKILTJC5YRW01"}} 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544572685657, "job": 1, "event": "recovery_finished"} 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x56174ca62a80 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: DB pointer 0x56174ca72000 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** DB Stats ** 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:56:12.788 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:56:12.789 INFO:journalctl@ceph.mon.b.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-10T05:56:12.790 INFO:journalctl@ceph.mon.b.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:56:12.791 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: using public_addr v2:172.21.15.150:0/0 -> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: starting mon.b rank -1 at public addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] at bind addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(???) e0 preinit fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).mds e1 new map 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout: e1 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T05:56:12.792 INFO:journalctl@ceph.mon.b.smithi150.stdout: legacy client fscid: -1 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout: No filesystems configured 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi139:172.21.15.139=a;smithi139:[v2:172.21.15.139:3301,v1:172.21.15.139:6790]=c;smithi150:172.21.15.150=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: Saving service mon spec with placement smithi139:172.21.15.139=a;smithi139:[v2:172.21.15.139:3301,v1:172.21.15.139:6790]=c;smithi150:172.21.15.150=b;count:3 2024-02-10T05:56:12.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: Deploying daemon mon.c on smithi139 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/2901703153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.a calling monitor election 2024-02-10T05:56:12.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.c calling monitor election 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: monmap e2: 2 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: fsmap 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:56:12.795 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mgrmap e13: y(active, since 36s) 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: overall HEALTH_OK 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: Deploying daemon mon.b on smithi150 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/220634138' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 ceph-mon[25582]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 bash[25520]: 51b65c6e04ce76ace48f94651a265b2a5ac31b0c55474b3633996e1cfe795eed 2024-02-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:12 smithi150 systemd[1]: Started Ceph mon.b for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:56:14.425 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:56:14.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:56:18.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:56:18.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: mon.c calling monitor election 2024-02-10T05:56:18.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: mon.a calling monitor election 2024-02-10T05:56:18.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: fsmap 2024-02-10T05:56:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:56:18.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: mgrmap e13: y(active, since 46s) 2024-02-10T05:56:18.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-10T05:56:18.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-10T05:56:18.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-10T05:56:18.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: mon.b (rank 2) addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] is down (out of quorum) 2024-02-10T05:56:18.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:18.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:18.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:18.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:18.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: mon.c calling monitor election 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: mon.a calling monitor election 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:18.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: fsmap 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: mgrmap e13: y(active, since 46s) 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: mon.b (rank 2) addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] is down (out of quorum) 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:18.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:19.129 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-10T05:56:19.129 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":3,"fsid":"ce44e742-c7d8-11ee-95b8-87774f69a715","modified":"2024-02-10T05:56:12.705021Z","created":"2024-02-10T05:54:34.334911Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3301","nonce":0},{"type":"v1","addr":"172.21.15.139:6790","nonce":0}]},"addr":"172.21.15.139:6790/0","public_addr":"172.21.15.139:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T05:56:19.131 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 3 2024-02-10T05:56:19.613 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T05:56:19.614 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T05:56:20.031 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.031 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:20.031 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T05:56:20.031 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.032 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T05:56:20.032 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.032 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.032 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.032 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:56:20.032 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:20.032 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:56:20.032 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:20.032 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: Reconfiguring daemon mon.a on smithi139 2024-02-10T05:56:20.032 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:20.032 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/1904636483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: Reconfiguring daemon mon.a on smithi139 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:20.033 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:19 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/1904636483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:56:20.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: mon.c calling monitor election 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: mon.a calling monitor election 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:56:20.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: fsmap 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: mgrmap e13: y(active, since 46s) 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: mon.b (rank 2) addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] is down (out of quorum) 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:20.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: Reconfiguring daemon mon.a on smithi139 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:20.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:19 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/1904636483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:56:21.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: mon.b calling monitor election 2024-02-10T05:56:21.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: mon.b calling monitor election 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: mon.c calling monitor election 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: mon.a calling monitor election 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: fsmap 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: mgrmap e13: y(active, since 48s) 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: Cluster is now healthy 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: overall HEALTH_OK 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:56:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:21.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: mon.b calling monitor election 2024-02-10T05:56:21.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: mon.b calling monitor election 2024-02-10T05:56:21.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: mon.c calling monitor election 2024-02-10T05:56:21.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: mon.a calling monitor election 2024-02-10T05:56:21.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T05:56:21.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T05:56:21.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: fsmap 2024-02-10T05:56:21.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:56:21.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: mgrmap e13: y(active, since 48s) 2024-02-10T05:56:21.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-10T05:56:21.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: Cluster is now healthy 2024-02-10T05:56:21.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: overall HEALTH_OK 2024-02-10T05:56:21.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:21.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:21.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:56:21.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:20 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: mon.b calling monitor election 2024-02-10T05:56:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: mon.b calling monitor election 2024-02-10T05:56:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: mon.c calling monitor election 2024-02-10T05:56:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: mon.a calling monitor election 2024-02-10T05:56:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T05:56:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T05:56:21.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: fsmap 2024-02-10T05:56:21.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:56:21.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: mgrmap e13: y(active, since 48s) 2024-02-10T05:56:21.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-10T05:56:21.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: Cluster is now healthy 2024-02-10T05:56:21.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: overall HEALTH_OK 2024-02-10T05:56:21.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:21.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:21.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:56:21.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:20 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:21.973 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:21 smithi139 ceph-mon[24727]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:56:21.973 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:21 smithi139 ceph-mon[24727]: Reconfiguring daemon mon.c on smithi139 2024-02-10T05:56:21.973 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:21 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:21.973 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:21 smithi139 ceph-mon[24727]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:21.973 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:21 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:21.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:21 smithi139 ceph-mon[29453]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:56:21.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:21 smithi139 ceph-mon[29453]: Reconfiguring daemon mon.c on smithi139 2024-02-10T05:56:21.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:21 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:21.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:21 smithi139 ceph-mon[29453]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:21.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:21 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:22.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:21 smithi150 ceph-mon[25582]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:56:22.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:21 smithi150 ceph-mon[25582]: Reconfiguring daemon mon.c on smithi139 2024-02-10T05:56:22.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:21 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:56:22.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:21 smithi150 ceph-mon[25582]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:22.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:21 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:22.677 INFO:teuthology.orchestra.run.smithi139.stdout:# minimal ceph.conf for ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:56:22.677 INFO:teuthology.orchestra.run.smithi139.stdout:[global] 2024-02-10T05:56:22.677 INFO:teuthology.orchestra.run.smithi139.stdout: fsid = ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:56:22.677 INFO:teuthology.orchestra.run.smithi139.stdout: mon_host = [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-02-10T05:56:22.903 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:22 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/58407524' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:22.903 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:22 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/58407524' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:22.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:22 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/58407524' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:23.562 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T05:56:23.562 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-02-10T05:56:23.562 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T05:56:23.591 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-02-10T05:56:23.592 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:23.659 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-10T05:56:23.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T05:56:23.688 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-10T05:56:23.688 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:23.755 INFO:tasks.cephadm:Adding mgr.y on smithi139 2024-02-10T05:56:23.755 INFO:tasks.cephadm:Adding mgr.x on smithi150 2024-02-10T05:56:23.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply mgr '2;smithi139=y;smithi150=x' 2024-02-10T05:56:24.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:24.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[29453]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:56:24.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:24.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:56:24.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:24.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[29453]: Reconfiguring daemon mon.b on smithi150 2024-02-10T05:56:24.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[29453]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[24727]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:56:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:56:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[24727]: Reconfiguring daemon mon.b on smithi150 2024-02-10T05:56:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:23 smithi139 ceph-mon[24727]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:24.212 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:23 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:24.212 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:23 smithi150 ceph-mon[25582]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:56:24.212 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:23 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:56:24.212 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:23 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:56:24.212 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:23 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:24.212 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:23 smithi150 ceph-mon[25582]: Reconfiguring daemon mon.b on smithi150 2024-02-10T05:56:24.212 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:23 smithi150 ceph-mon[25582]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:25.327 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:25 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:25.327 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:25 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:25.328 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:25 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:25.328 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:25 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:25.328 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:25 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:25.328 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:25 smithi150 ceph-mon[25582]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:25.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:25.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:25.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:25.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:25.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:25.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[29453]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:25.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:25.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:25.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:25.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:25.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:25.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:25 smithi139 ceph-mon[24727]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:25.628 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled mgr update... 2024-02-10T05:56:26.238 DEBUG:teuthology.orchestra.run.smithi150:mgr.x> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.x.service 2024-02-10T05:56:26.240 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T05:56:26.240 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-02-10T05:56:26.240 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:56:26.260 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T05:56:26.260 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 2024-02-10T05:56:26.318 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T05:56:26.318 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:56:26.318 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-10T05:56:26.318 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:56:26.318 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:56:26.318 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-02-10 05:56:01.657850122 +0000 2024-02-10T05:56:26.318 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-02-10 05:53:44.944953106 +0000 2024-02-10T05:56:26.318 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-02-10 05:53:44.944953106 +0000 2024-02-10T05:56:26.319 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-02-10 05:53:44.944953106 +0000 2024-02-10T05:56:26.319 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T05:56:26.390 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-02-10T05:56:26.390 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-02-10T05:56:26.390 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000265768 s, 1.9 MB/s 2024-02-10T05:56:26.391 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T05:56:26.450 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 2024-02-10T05:56:26.508 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T05:56:26.508 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:56:26.508 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-10T05:56:26.508 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:56:26.508 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:56:26.508 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-02-10 05:56:01.657850122 +0000 2024-02-10T05:56:26.508 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-02-10 05:53:45.284945834 +0000 2024-02-10T05:56:26.508 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-02-10 05:53:45.284945834 +0000 2024-02-10T05:56:26.508 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-02-10 05:53:45.284945834 +0000 2024-02-10T05:56:26.509 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T05:56:26.576 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-02-10T05:56:26.576 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-02-10T05:56:26.576 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.00023379 s, 2.2 MB/s 2024-02-10T05:56:26.578 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T05:56:26.636 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 2024-02-10T05:56:26.692 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T05:56:26.693 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:56:26.693 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-10T05:56:26.693 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:56:26.693 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:56:26.693 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-02-10 05:56:01.657850122 +0000 2024-02-10T05:56:26.693 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-02-10 05:53:45.613938798 +0000 2024-02-10T05:56:26.693 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-02-10 05:53:45.613938798 +0000 2024-02-10T05:56:26.693 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-02-10 05:53:45.613938798 +0000 2024-02-10T05:56:26.693 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T05:56:26.736 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi139=y;smithi150=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:26.736 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: Saving service mgr spec with placement smithi139=y;smithi150=x;count:2 2024-02-10T05:56:26.736 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:26.737 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:26.737 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:26.737 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:26.737 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:26.737 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:56:26.737 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:56:26.737 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:56:26.738 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:26.738 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:26 smithi150 ceph-mon[25582]: Deploying daemon mgr.x on smithi150 2024-02-10T05:56:26.759 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi139=y;smithi150=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:26.759 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: Saving service mgr spec with placement smithi139=y;smithi150=x;count:2 2024-02-10T05:56:26.759 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:26.759 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:26.759 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:26.759 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:26.759 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:26.759 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:56:26.760 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:56:26.760 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:56:26.760 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:26.760 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[24727]: Deploying daemon mgr.x on smithi150 2024-02-10T05:56:26.760 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi139=y;smithi150=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:26.760 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: Saving service mgr spec with placement smithi139=y;smithi150=x;count:2 2024-02-10T05:56:26.760 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:26.760 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:26.760 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:26.760 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:26.760 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:26.761 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:56:26.761 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:56:26.761 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:56:26.761 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:26.761 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:26 smithi139 ceph-mon[29453]: Deploying daemon mgr.x on smithi150 2024-02-10T05:56:26.761 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-02-10T05:56:26.761 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-02-10T05:56:26.761 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000224315 s, 2.3 MB/s 2024-02-10T05:56:26.763 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T05:56:26.821 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 2024-02-10T05:56:26.877 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T05:56:26.877 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:56:26.877 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-10T05:56:26.877 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:56:26.877 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:56:26.877 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-02-10 05:56:01.658850101 +0000 2024-02-10T05:56:26.877 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-02-10 05:53:45.958931419 +0000 2024-02-10T05:56:26.877 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-02-10 05:53:45.958931419 +0000 2024-02-10T05:56:26.878 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-02-10 05:53:45.958931419 +0000 2024-02-10T05:56:26.878 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T05:56:26.944 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-02-10T05:56:26.944 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-02-10T05:56:26.944 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000179192 s, 2.9 MB/s 2024-02-10T05:56:26.946 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T05:56:27.002 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-10T05:56:27.002 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:56:27.018 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T05:56:27.018 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-02-10T05:56:27.073 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T05:56:27.111 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:56:27.111 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-10T05:56:27.111 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:56:27.111 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:56:27.111 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-10 05:56:10.984123567 +0000 2024-02-10T05:56:27.111 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-10 05:53:44.775000498 +0000 2024-02-10T05:56:27.111 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-10 05:53:44.775000498 +0000 2024-02-10T05:56:27.111 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-10 05:53:44.775000498 +0000 2024-02-10T05:56:27.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T05:56:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-10T05:56:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-10T05:56:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000246305 s, 2.1 MB/s 2024-02-10T05:56:27.142 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T05:56:27.198 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-02-10T05:56:27.254 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T05:56:27.254 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:56:27.254 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-10T05:56:27.254 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:56:27.254 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:56:27.254 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-10 05:56:10.985123547 +0000 2024-02-10T05:56:27.254 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-10 05:53:45.109993964 +0000 2024-02-10T05:56:27.254 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-10 05:53:45.109993964 +0000 2024-02-10T05:56:27.254 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-10 05:53:45.109993964 +0000 2024-02-10T05:56:27.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T05:56:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-10T05:56:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-10T05:56:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000232422 s, 2.2 MB/s 2024-02-10T05:56:27.321 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T05:56:27.377 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-02-10T05:56:27.432 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T05:56:27.432 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:56:27.432 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-10T05:56:27.432 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:56:27.432 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:56:27.432 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-10 05:56:10.985123547 +0000 2024-02-10T05:56:27.433 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-10 05:53:45.450987312 +0000 2024-02-10T05:56:27.433 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-10 05:53:45.450987312 +0000 2024-02-10T05:56:27.433 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-10 05:53:45.450987312 +0000 2024-02-10T05:56:27.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T05:56:27.564 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-10T05:56:27.564 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-10T05:56:27.564 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000199095 s, 2.6 MB/s 2024-02-10T05:56:27.565 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T05:56:27.580 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-02-10T05:56:27.643 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T05:56:27.643 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:56:27.643 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-10T05:56:27.643 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:56:27.643 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:56:27.643 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-10 05:56:27.524803669 +0000 2024-02-10T05:56:27.643 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-10 05:53:45.979976994 +0000 2024-02-10T05:56:27.643 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-10 05:53:45.979976994 +0000 2024-02-10T05:56:27.644 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-10 05:53:45.979976994 +0000 2024-02-10T05:56:27.644 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T05:56:27.707 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:27 smithi150 ceph-mon[25582]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:27.742 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-10T05:56:27.742 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-10T05:56:27.742 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.00017038 s, 3.0 MB/s 2024-02-10T05:56:27.744 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T05:56:27.802 INFO:tasks.cephadm:Deploying osd.0 on smithi139 with /dev/vg_nvme/lv_4... 2024-02-10T05:56:27.802 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T05:56:27.976 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:27 smithi150 podman[26748]: 2024-02-10 05:56:27.880054661 +0000 UTC m=+0.018891816 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-02-10T05:56:28.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:27 smithi139 ceph-mon[24727]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:28.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:27 smithi139 ceph-mon[29453]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:28.244 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:28 smithi150 podman[26748]: 2024-02-10 05:56:28.145121274 +0000 UTC m=+0.283958424 container init 62310c7d2542ae5d8abfb6d454b68f29af33baeaaba327e93662f2ebdc1c722f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, vcs-type=git, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, version=8, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8) 2024-02-10T05:56:28.244 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:28 smithi150 podman[26748]: 2024-02-10 05:56:28.148050526 +0000 UTC m=+0.286887669 container start 62310c7d2542ae5d8abfb6d454b68f29af33baeaaba327e93662f2ebdc1c722f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.component=centos-stream-container, RELEASE=HEAD, maintainer=Guillaume Abrioux , ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public) 2024-02-10T05:56:28.245 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:28 smithi150 ceph-mgr[26767]: -- 172.21.15.150:0/3215702293 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556da089c340 con 0x556da0875000 2024-02-10T05:56:28.245 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:28 smithi150 bash[26748]: 62310c7d2542ae5d8abfb6d454b68f29af33baeaaba327e93662f2ebdc1c722f 2024-02-10T05:56:28.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:28 smithi150 systemd[1]: Started Ceph mgr.x for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:56:28.961 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:28 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:28.652+0000 7f51dd893000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:56:28.961 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:28 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:28.745+0000 7f51dd893000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:56:29.528 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:29.357+0000 7f51dd893000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:56:29.528 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:29.446+0000 7f51dd893000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:56:29.528 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:29.527+0000 7f51dd893000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:56:29.528 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:29.528 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:29.528 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:29.528 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:29.528 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-mon[25582]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:29.528 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:29.529 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:29.529 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:56:29.529 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:56:29.529 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:29.549 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:29.549 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:29.549 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:29.549 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:29.549 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[29453]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:29.549 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:29.549 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[24727]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:56:29.550 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:29.853 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T05:56:29.853 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T05:56:29.853 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-02-10T05:56:29.853 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-02-10T05:56:29.853 INFO:teuthology.orchestra.run.smithi139.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113107 s, 927 MB/s 2024-02-10T05:56:29.853 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 2024-02-10T05:56:29.853 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-02-10T05:56:29.863 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:29.675+0000 7f51dd893000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:56:29.864 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:29.755+0000 7f51dd893000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:56:29.864 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:29.863+0000 7f51dd893000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:56:30.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:29 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:29.945+0000 7f51dd893000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:56:30.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:30.107+0000 7f51dd893000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:56:30.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:30.192+0000 7f51dd893000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:56:30.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:30 smithi139 ceph-mon[24727]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:56:30.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:30 smithi139 ceph-mon[24727]: Reconfiguring daemon mgr.y on smithi139 2024-02-10T05:56:30.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:30 smithi139 ceph-mon[29453]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:56:30.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:30 smithi139 ceph-mon[29453]: Reconfiguring daemon mgr.y on smithi139 2024-02-10T05:56:31.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:30 smithi150 ceph-mon[25582]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:56:31.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:30 smithi150 ceph-mon[25582]: Reconfiguring daemon mgr.y on smithi139 2024-02-10T05:56:31.327 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi139:vg_nvme/lv_4 2024-02-10T05:56:31.532 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:31.216+0000 7f51dd893000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:56:31.532 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:31.343+0000 7f51dd893000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:56:31.532 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:31.531+0000 7f51dd893000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:56:31.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:31.617+0000 7f51dd893000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:56:31.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:31.751+0000 7f51dd893000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:56:32.288 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:32 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:32.068+0000 7f51dd893000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:56:32.288 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:32.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:32.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:32.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:32.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:32.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-mon[25582]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:32.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:56:32.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:56:32.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[29453]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[24727]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:56:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:56:32.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:32.539 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:32 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:32.288+0000 7f51dd893000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:56:32.539 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:32 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:32.371+0000 7f51dd893000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:56:32.539 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:32 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:32.538+0000 7f51dd893000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:56:33.535 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:33 smithi139 ceph-mon[24727]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:33.535 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:33 smithi139 ceph-mon[24727]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:33.535 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:33 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:33.535 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:33 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:33.535 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:33 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:33.535 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:33 smithi139 ceph-mon[29453]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:33.535 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:33 smithi139 ceph-mon[29453]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:33.535 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:33 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:33.536 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:33 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:33.536 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:33 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:33.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:33.447+0000 7f51dd893000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:56:33.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:33 smithi150 ceph-mon[25582]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:33.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:33 smithi150 ceph-mon[25582]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:33.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:33 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:33.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:33 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:33.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:33 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:34.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:33.738+0000 7f51dd893000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:56:34.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:33.832+0000 7f51dd893000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:56:34.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:56:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:56:33.979+0000 7f51dd893000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:56:35.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:34 smithi139 ceph-mon[24727]: Standby manager daemon x started 2024-02-10T05:56:35.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:34 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:56:35.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:34 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:56:35.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:34 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:56:35.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:34 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:56:35.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:34 smithi139 ceph-mon[29453]: Standby manager daemon x started 2024-02-10T05:56:35.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:34 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:56:35.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:34 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:56:35.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:34 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:56:35.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:34 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:56:35.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:34 smithi150 ceph-mon[25582]: Standby manager daemon x started 2024-02-10T05:56:35.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:34 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:56:35.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:34 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:56:35.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:34 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:56:35.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:34 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/1973013873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:56:36.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[24727]: mgrmap e14: y(active, since 63s), standbys: x 2024-02-10T05:56:36.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:56:36.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[24727]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3355191973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f"}]: dispatch 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f"}]: dispatch 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f"}]': finished 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[24727]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[29453]: mgrmap e14: y(active, since 63s), standbys: x 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[29453]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3355191973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f"}]: dispatch 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f"}]: dispatch 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f"}]': finished 2024-02-10T05:56:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[29453]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:56:36.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:35 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:35 smithi150 ceph-mon[25582]: mgrmap e14: y(active, since 63s), standbys: x 2024-02-10T05:56:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:35 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:56:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:35 smithi150 ceph-mon[25582]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:35 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3355191973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f"}]: dispatch 2024-02-10T05:56:36.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:35 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f"}]: dispatch 2024-02-10T05:56:36.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:35 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f"}]': finished 2024-02-10T05:56:36.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:35 smithi150 ceph-mon[25582]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:56:36.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:35 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:37.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:36 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/400556079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:56:37.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:36 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/400556079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:56:37.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:36 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/400556079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:56:38.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:37 smithi139 ceph-mon[24727]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:38.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:37 smithi139 ceph-mon[29453]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:38.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:37 smithi150 ceph-mon[25582]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:39.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:39 smithi139 ceph-mon[24727]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:39.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:39 smithi139 ceph-mon[29453]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:40.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:39 smithi150 ceph-mon[25582]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:42.430 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:42 smithi139 ceph-mon[24727]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:42.431 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:42 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:56:42.431 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:42 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:42.431 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:42 smithi139 ceph-mon[29453]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:42.431 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:42 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:56:42.431 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:42 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:42.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:42 smithi150 ceph-mon[25582]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:42.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:42 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:56:42.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:42 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:43.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:43 smithi139 ceph-mon[29453]: Deploying daemon osd.0 on smithi139 2024-02-10T05:56:43.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:43 smithi139 ceph-mon[24727]: Deploying daemon osd.0 on smithi139 2024-02-10T05:56:43.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:43 smithi150 ceph-mon[25582]: Deploying daemon osd.0 on smithi139 2024-02-10T05:56:44.486 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:44 smithi139 ceph-mon[24727]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:44.487 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:44 smithi139 ceph-mon[29453]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:44 smithi150 ceph-mon[25582]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:46.448 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:46 smithi139 ceph-mon[24727]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:46.448 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:46 smithi139 ceph-mon[29453]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:46 smithi150 ceph-mon[25582]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:48.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:47 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:48.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:47 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:48.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:47 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:48.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:47 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:48.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:47 smithi139 ceph-mon[29453]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:48.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:48.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:48.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:48.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:48.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:47 smithi139 ceph-mon[24727]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:48.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:47 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:48.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:47 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:48.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:47 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:48.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:47 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:48.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:47 smithi150 ceph-mon[25582]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:49.904 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:49 smithi139 ceph-mon[24727]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:49.904 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:49 smithi139 ceph-mon[29453]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:49.980 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 0 on host 'smithi139' 2024-02-10T05:56:50.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:49 smithi150 ceph-mon[25582]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:50.589 DEBUG:teuthology.orchestra.run.smithi139:osd.0> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T05:56:50.591 INFO:tasks.cephadm:Deploying osd.1 on smithi139 with /dev/vg_nvme/lv_3... 2024-02-10T05:56:50.591 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T05:56:50.859 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:50 smithi139 ceph-mon[24727]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:56:50.859 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:50 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:50.859 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:50 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:50.859 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:50 smithi139 ceph-mon[29453]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:56:50.859 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:50 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:50.859 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:50 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:51.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:50 smithi150 ceph-mon[25582]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:56:51.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:50 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:51.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:50 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-mon[29453]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:56:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-mon[29453]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:56:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-mon[29453]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:56:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-mon[29453]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-mon[24727]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:56:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-mon[24727]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:56:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-mon[24727]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:56:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-mon[24727]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:52.087 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[32312]: 2024-02-10T05:56:51.777+0000 7f6cf3d54700 -1 osd.0 0 waiting for initial osdmap 2024-02-10T05:56:52.087 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 05:56:51 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[32312]: 2024-02-10T05:56:51.780+0000 7f6cedeea700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:56:52.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:51 smithi150 ceph-mon[25582]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:56:52.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:51 smithi150 ceph-mon[25582]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:56:52.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:51 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:52.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:51 smithi150 ceph-mon[25582]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:56:52.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:51 smithi150 ceph-mon[25582]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:52.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:52 smithi139 ceph-mon[24727]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:56:52.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:52 smithi139 ceph-mon[24727]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:56:52.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:52 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:52.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:52 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:52.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:52 smithi139 ceph-mon[29453]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:56:52.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:52 smithi139 ceph-mon[29453]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:56:52.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:52 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:52.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:52 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:53.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:52 smithi150 ceph-mon[25582]: from='osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:56:53.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:52 smithi150 ceph-mon[25582]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:56:53.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:52 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:53.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:52 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:53.978 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T05:56:53.978 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T05:56:53.978 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-02-10T05:56:53.978 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-02-10T05:56:53.979 INFO:teuthology.orchestra.run.smithi139.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012278 s, 854 MB/s 2024-02-10T05:56:53.979 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-02-10T05:56:53.979 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[24727]: purged_snaps scrub starts 2024-02-10T05:56:53.979 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[24727]: purged_snaps scrub ok 2024-02-10T05:56:53.979 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[24727]: osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395] boot 2024-02-10T05:56:53.979 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[24727]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:56:53.979 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:53.979 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[24727]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:53.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[29453]: purged_snaps scrub starts 2024-02-10T05:56:53.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[29453]: purged_snaps scrub ok 2024-02-10T05:56:53.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[29453]: osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395] boot 2024-02-10T05:56:53.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[29453]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:56:53.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:53.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:53 smithi139 ceph-mon[29453]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:53 smithi150 ceph-mon[25582]: purged_snaps scrub starts 2024-02-10T05:56:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:53 smithi150 ceph-mon[25582]: purged_snaps scrub ok 2024-02-10T05:56:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:53 smithi150 ceph-mon[25582]: osd.0 [v2:172.21.15.139:6802/3875590395,v1:172.21.15.139:6803/3875590395] boot 2024-02-10T05:56:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:53 smithi150 ceph-mon[25582]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:56:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:53 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:56:54.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:53 smithi150 ceph-mon[25582]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:56:55.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[24727]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:56:55.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:55.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:55.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:55.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:55.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:55.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:55.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:55.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[29453]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:56:55.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:55.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:55.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:55.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:55.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:55.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:55.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:54 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:55.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:54 smithi150 ceph-mon[25582]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:56:55.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:54 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:55.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:54 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:55.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:54 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:55.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:54 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:55.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:54 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:55.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:54 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:55.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:54 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:55.142 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi139:vg_nvme/lv_3 2024-02-10T05:56:56.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:55 smithi139 ceph-mon[24727]: Detected new or changed devices on smithi139 2024-02-10T05:56:56.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:55 smithi139 ceph-mon[24727]: Adjusting osd_memory_target on smithi139 to 16064M 2024-02-10T05:56:56.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:55 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:56.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:55 smithi139 ceph-mon[24727]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:56:56.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:55 smithi139 ceph-mon[29453]: Detected new or changed devices on smithi139 2024-02-10T05:56:56.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:55 smithi139 ceph-mon[29453]: Adjusting osd_memory_target on smithi139 to 16064M 2024-02-10T05:56:56.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:55 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:56.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:55 smithi139 ceph-mon[29453]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:56:56.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:55 smithi150 ceph-mon[25582]: Detected new or changed devices on smithi139 2024-02-10T05:56:56.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:55 smithi150 ceph-mon[25582]: Adjusting osd_memory_target on smithi139 to 16064M 2024-02-10T05:56:56.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:55 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:56:56.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:55 smithi150 ceph-mon[25582]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:56:57.507 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:57 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:57.507 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:57 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:57.507 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:57 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:57.507 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:57 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:57.507 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:57 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:57.507 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:57 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:57 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:57 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:57 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:58.429 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:58 smithi139 ceph-mon[24727]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:56:58.429 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:58 smithi139 ceph-mon[24727]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:58.429 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:58 smithi139 ceph-mon[29453]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:56:58.429 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:58 smithi139 ceph-mon[29453]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:58 smithi150 ceph-mon[25582]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:56:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:58 smithi150 ceph-mon[25582]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:00.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[24727]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:00.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/139498351' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39bbeb2-0fd3-4584-bde1-2578c452924e"}]: dispatch 2024-02-10T05:57:00.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39bbeb2-0fd3-4584-bde1-2578c452924e"}]: dispatch 2024-02-10T05:57:00.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a39bbeb2-0fd3-4584-bde1-2578c452924e"}]': finished 2024-02-10T05:57:00.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[24727]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:57:00.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[29453]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/139498351' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39bbeb2-0fd3-4584-bde1-2578c452924e"}]: dispatch 2024-02-10T05:57:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39bbeb2-0fd3-4584-bde1-2578c452924e"}]: dispatch 2024-02-10T05:57:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a39bbeb2-0fd3-4584-bde1-2578c452924e"}]': finished 2024-02-10T05:57:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[29453]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:57:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:56:59 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:59 smithi150 ceph-mon[25582]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:59 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/139498351' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39bbeb2-0fd3-4584-bde1-2578c452924e"}]: dispatch 2024-02-10T05:57:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:59 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39bbeb2-0fd3-4584-bde1-2578c452924e"}]: dispatch 2024-02-10T05:57:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:59 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a39bbeb2-0fd3-4584-bde1-2578c452924e"}]': finished 2024-02-10T05:57:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:59 smithi150 ceph-mon[25582]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:57:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:56:59 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:01.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:00 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2395696645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:01.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:00 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2395696645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:01.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:00 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2395696645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:02.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:01 smithi139 ceph-mon[24727]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:02.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:01 smithi139 ceph-mon[29453]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:02.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:01 smithi150 ceph-mon[25582]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:04.184 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:04 smithi139 ceph-mon[24727]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:04.184 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:04 smithi139 ceph-mon[29453]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:04 smithi150 ceph-mon[25582]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:06.251 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:06 smithi139 ceph-mon[24727]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:06.252 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:57:06.252 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:06.252 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:06 smithi139 ceph-mon[29453]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:06.252 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:57:06.252 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:06 smithi150 ceph-mon[25582]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:57:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:07.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:07 smithi139 ceph-mon[24727]: Deploying daemon osd.1 on smithi139 2024-02-10T05:57:07.497 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:07 smithi139 ceph-mon[29453]: Deploying daemon osd.1 on smithi139 2024-02-10T05:57:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:07 smithi150 ceph-mon[25582]: Deploying daemon osd.1 on smithi139 2024-02-10T05:57:08.328 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:08 smithi139 ceph-mon[24727]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:08.328 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:08 smithi139 ceph-mon[29453]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:08 smithi150 ceph-mon[25582]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:10.009 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:09 smithi139 ceph-mon[24727]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:10.009 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:09 smithi139 ceph-mon[29453]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:09 smithi150 ceph-mon[25582]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:11.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:11 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:11.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:11 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:11.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:11 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:11.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:11 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:11.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:11 smithi150 ceph-mon[25582]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:12.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:11 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:12.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:11 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:12.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:11 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:12.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:11 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:12.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:11 smithi139 ceph-mon[24727]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:12.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:11 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:12.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:11 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:12.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:11 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:12.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:11 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:12.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:11 smithi139 ceph-mon[29453]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:13.933 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 1 on host 'smithi139' 2024-02-10T05:57:14.257 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:14.257 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:14.257 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[24727]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:14.257 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[24727]: from='osd.1 [v2:172.21.15.139:6810/2796500287,v1:172.21.15.139:6811/2796500287]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:57:14.257 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[24727]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:57:14.257 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:14.257 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:14.257 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:14.258 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:14.258 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:14.258 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:14.258 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[29453]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:14.258 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[29453]: from='osd.1 [v2:172.21.15.139:6810/2796500287,v1:172.21.15.139:6811/2796500287]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:57:14.258 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[29453]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:57:14.258 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:14.258 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:14.258 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:14.258 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:14 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:14.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:14 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:14.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:14 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:14.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:14 smithi150 ceph-mon[25582]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:14.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:14 smithi150 ceph-mon[25582]: from='osd.1 [v2:172.21.15.139:6810/2796500287,v1:172.21.15.139:6811/2796500287]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:57:14.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:14 smithi150 ceph-mon[25582]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:57:14.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:14 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:14.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:14 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:14.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:14 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:14.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:14 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:14.705 DEBUG:teuthology.orchestra.run.smithi139:osd.1> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T05:57:14.707 INFO:tasks.cephadm:Deploying osd.2 on smithi139 with /dev/vg_nvme/lv_2... 2024-02-10T05:57:14.708 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T05:57:15.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-mon[29453]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:57:15.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-mon[29453]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:57:15.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:15.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-mon[29453]: from='osd.1 [v2:172.21.15.139:6810/2796500287,v1:172.21.15.139:6811/2796500287]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:57:15.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-mon[29453]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:57:15.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-mon[24727]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:57:15.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-mon[24727]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:57:15.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:15.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-mon[24727]: from='osd.1 [v2:172.21.15.139:6810/2796500287,v1:172.21.15.139:6811/2796500287]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:57:15.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-mon[24727]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:57:15.342 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[35284]: 2024-02-10T05:57:15.108+0000 7fb952392700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T05:57:15.342 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 05:57:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[35284]: 2024-02-10T05:57:15.110+0000 7fb94dd2b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:57:15.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:15 smithi150 ceph-mon[25582]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:57:15.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:15 smithi150 ceph-mon[25582]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:57:15.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:15 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:15.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:15 smithi150 ceph-mon[25582]: from='osd.1 [v2:172.21.15.139:6810/2796500287,v1:172.21.15.139:6811/2796500287]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:57:15.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:15 smithi150 ceph-mon[25582]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:57:16.134 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:16 smithi139 ceph-mon[24727]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:57:16.134 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:16 smithi139 ceph-mon[24727]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:57:16.134 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:16 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:16.134 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:16 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:16.134 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:16 smithi139 ceph-mon[24727]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:16.135 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:16 smithi139 ceph-mon[29453]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:57:16.135 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:16 smithi139 ceph-mon[29453]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:57:16.135 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:16 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:16.135 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:16 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:16.135 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:16 smithi139 ceph-mon[29453]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:16.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:16 smithi150 ceph-mon[25582]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:57:16.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:16 smithi150 ceph-mon[25582]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:57:16.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:16 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:16.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:16 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:16.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:16 smithi150 ceph-mon[25582]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:57:17.373 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:17 smithi139 ceph-mon[24727]: purged_snaps scrub starts 2024-02-10T05:57:17.373 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:17 smithi139 ceph-mon[24727]: purged_snaps scrub ok 2024-02-10T05:57:17.373 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:17 smithi139 ceph-mon[24727]: osd.1 [v2:172.21.15.139:6810/2796500287,v1:172.21.15.139:6811/2796500287] boot 2024-02-10T05:57:17.373 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:17 smithi139 ceph-mon[24727]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:57:17.373 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:17 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:17.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:17 smithi139 ceph-mon[29453]: purged_snaps scrub starts 2024-02-10T05:57:17.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:17 smithi139 ceph-mon[29453]: purged_snaps scrub ok 2024-02-10T05:57:17.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:17 smithi139 ceph-mon[29453]: osd.1 [v2:172.21.15.139:6810/2796500287,v1:172.21.15.139:6811/2796500287] boot 2024-02-10T05:57:17.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:17 smithi139 ceph-mon[29453]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:57:17.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:17 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:17.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:17 smithi150 ceph-mon[25582]: purged_snaps scrub starts 2024-02-10T05:57:17.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:17 smithi150 ceph-mon[25582]: purged_snaps scrub ok 2024-02-10T05:57:17.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:17 smithi150 ceph-mon[25582]: osd.1 [v2:172.21.15.139:6810/2796500287,v1:172.21.15.139:6811/2796500287] boot 2024-02-10T05:57:17.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:17 smithi150 ceph-mon[25582]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:57:17.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:17 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:57:18.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[29453]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:57:18.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[29453]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:18.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:18.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[24727]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:57:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[24727]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:18 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:18.569 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T05:57:18.569 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T05:57:18.569 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-02-10T05:57:18.569 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-02-10T05:57:18.569 INFO:teuthology.orchestra.run.smithi139.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127826 s, 820 MB/s 2024-02-10T05:57:18.569 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 2024-02-10T05:57:18.570 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-02-10T05:57:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:18 smithi150 ceph-mon[25582]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:57:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:18 smithi150 ceph-mon[25582]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:18 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:18 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:18 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:18 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:18 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:19.166 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:19 smithi139 ceph-mon[24727]: Detected new or changed devices on smithi139 2024-02-10T05:57:19.166 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:19 smithi139 ceph-mon[24727]: Adjusting osd_memory_target on smithi139 to 8032M 2024-02-10T05:57:19.166 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:19 smithi139 ceph-mon[29453]: Detected new or changed devices on smithi139 2024-02-10T05:57:19.166 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:19 smithi139 ceph-mon[29453]: Adjusting osd_memory_target on smithi139 to 8032M 2024-02-10T05:57:19.189 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi139:vg_nvme/lv_2 2024-02-10T05:57:19.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:19 smithi150 ceph-mon[25582]: Detected new or changed devices on smithi139 2024-02-10T05:57:19.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:19 smithi150 ceph-mon[25582]: Adjusting osd_memory_target on smithi139 to 8032M 2024-02-10T05:57:20.396 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:20 smithi139 ceph-mon[24727]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:20.396 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:20 smithi139 ceph-mon[29453]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:20.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:20 smithi150 ceph-mon[25582]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:21 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:57:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:21 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:57:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:21 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:21 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:57:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:21 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:57:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:21 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:21.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:21 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:57:21.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:21 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:57:21.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:21 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:22.479 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:22 smithi139 ceph-mon[24727]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:22.480 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:22 smithi139 ceph-mon[24727]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:22.480 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:22 smithi139 ceph-mon[29453]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:22.480 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:22 smithi139 ceph-mon[29453]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:22 smithi150 ceph-mon[25582]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:22 smithi150 ceph-mon[25582]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[24727]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:24.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3677541693' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f801d19-78c0-40e6-8f36-efd49380be35"}]: dispatch 2024-02-10T05:57:24.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3677541693' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f801d19-78c0-40e6-8f36-efd49380be35"}]': finished 2024-02-10T05:57:24.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[24727]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:57:24.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:24.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2045239506' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[29453]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3677541693' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f801d19-78c0-40e6-8f36-efd49380be35"}]: dispatch 2024-02-10T05:57:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3677541693' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f801d19-78c0-40e6-8f36-efd49380be35"}]': finished 2024-02-10T05:57:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[29453]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:57:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:24.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:24 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2045239506' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:24.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:24 smithi150 ceph-mon[25582]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:24.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:24 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3677541693' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f801d19-78c0-40e6-8f36-efd49380be35"}]: dispatch 2024-02-10T05:57:24.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:24 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3677541693' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f801d19-78c0-40e6-8f36-efd49380be35"}]': finished 2024-02-10T05:57:24.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:24 smithi150 ceph-mon[25582]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:57:24.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:24 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:24.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:24 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2045239506' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:26.555 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:26 smithi139 ceph-mon[24727]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:26.555 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:26 smithi139 ceph-mon[29453]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:26.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:26 smithi150 ceph-mon[25582]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:28.318 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:28 smithi139 ceph-mon[24727]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:28.318 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:28 smithi139 ceph-mon[29453]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:28.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:28 smithi150 ceph-mon[25582]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:30.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:29 smithi139 ceph-mon[24727]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:30.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:57:30.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:30.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:29 smithi139 ceph-mon[29453]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:30.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:57:30.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:30.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:29 smithi150 ceph-mon[25582]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:30.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:57:30.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:31.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:30 smithi139 ceph-mon[24727]: Deploying daemon osd.2 on smithi139 2024-02-10T05:57:31.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:30 smithi139 ceph-mon[29453]: Deploying daemon osd.2 on smithi139 2024-02-10T05:57:31.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:30 smithi150 ceph-mon[25582]: Deploying daemon osd.2 on smithi139 2024-02-10T05:57:32.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:31 smithi139 ceph-mon[24727]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:32.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:57:32.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:57:32.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:31 smithi139 ceph-mon[29453]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:32.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:57:32.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:57:32.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:31 smithi150 ceph-mon[25582]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:32.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:57:32.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:57:34.476 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:34 smithi139 ceph-mon[24727]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:34.477 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:34 smithi139 ceph-mon[29453]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:34.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:34 smithi150 ceph-mon[25582]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:36.062 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:35 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:36.062 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:35 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:36.062 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:35 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:36.062 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:35 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:36.062 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:35 smithi139 ceph-mon[24727]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:36.062 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:35 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:36.063 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:35 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:36.063 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:35 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:36.063 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:35 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:36.063 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:35 smithi139 ceph-mon[29453]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:35 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:35 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:35 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:35 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:35 smithi150 ceph-mon[25582]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:37.587 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 2 on host 'smithi139' 2024-02-10T05:57:38.507 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:38 smithi139 ceph-mon[24727]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:38.507 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:38 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:38.508 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:38 smithi139 ceph-mon[24727]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:57:38.508 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:38 smithi139 ceph-mon[29453]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:38.508 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:38 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:38.508 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:38 smithi139 ceph-mon[29453]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:57:38.530 DEBUG:teuthology.orchestra.run.smithi139:osd.2> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T05:57:38.533 INFO:tasks.cephadm:Deploying osd.3 on smithi139 with /dev/vg_nvme/lv_1... 2024-02-10T05:57:38.533 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T05:57:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:38 smithi150 ceph-mon[25582]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:38 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:38 smithi150 ceph-mon[25582]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:57:39.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:39 smithi150 ceph-mon[25582]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:57:39.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:39 smithi150 ceph-mon[25582]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:57:39.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:39 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:39.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:39 smithi150 ceph-mon[25582]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:57:39.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:39 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:39.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:39 smithi150 ceph-mon[25582]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:39.911 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[24727]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:57:39.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[24727]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:57:39.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:39.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[24727]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:57:39.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:39.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[24727]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:39.912 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[29453]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:57:39.912 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[29453]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:57:39.912 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:39.912 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[29453]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:57:39.912 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:39.913 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-mon[29453]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:39.913 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[38210]: 2024-02-10T05:57:39.595+0000 7f3c6aafa700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T05:57:39.913 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 05:57:39 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[38210]: 2024-02-10T05:57:39.598+0000 7f3c67495700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:57:40.621 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T05:57:40.621 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T05:57:40.622 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-02-10T05:57:40.622 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-02-10T05:57:40.622 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126608 s, 828 MB/s 2024-02-10T05:57:40.622 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-02-10T05:57:40.827 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:40 smithi139 ceph-mon[29453]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:57:40.827 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:40 smithi139 ceph-mon[29453]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:57:40.827 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:40 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:40.827 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:40 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:40.827 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:40 smithi139 ceph-mon[24727]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:57:40.827 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:40 smithi139 ceph-mon[24727]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:57:40.827 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:40 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:40.827 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:40 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:40.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:40 smithi150 ceph-mon[25582]: from='osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:57:40.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:40 smithi150 ceph-mon[25582]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:57:40.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:40 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:40.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:40 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:41.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:41 smithi150 ceph-mon[25582]: purged_snaps scrub starts 2024-02-10T05:57:41.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:41 smithi150 ceph-mon[25582]: purged_snaps scrub ok 2024-02-10T05:57:41.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:41 smithi150 ceph-mon[25582]: osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670] boot 2024-02-10T05:57:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:41 smithi150 ceph-mon[25582]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:57:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:41 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:41 smithi150 ceph-mon[25582]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:41 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T05:57:41.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[24727]: purged_snaps scrub starts 2024-02-10T05:57:41.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[24727]: purged_snaps scrub ok 2024-02-10T05:57:41.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[24727]: osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670] boot 2024-02-10T05:57:41.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[24727]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:57:41.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:41.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[24727]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:41.912 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T05:57:41.912 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[29453]: purged_snaps scrub starts 2024-02-10T05:57:41.912 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[29453]: purged_snaps scrub ok 2024-02-10T05:57:41.913 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[29453]: osd.2 [v2:172.21.15.139:6818/872620670,v1:172.21.15.139:6819/872620670] boot 2024-02-10T05:57:41.913 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[29453]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:57:41.913 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:57:41.913 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[29453]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:57:41.913 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:41 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T05:57:42.621 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi139:vg_nvme/lv_1 2024-02-10T05:57:42.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:42 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T05:57:42.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:42 smithi150 ceph-mon[25582]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:57:42.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:42 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:57:43.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:42 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T05:57:43.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:42 smithi139 ceph-mon[29453]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:57:43.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:42 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:57:43.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:42 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T05:57:43.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:42 smithi139 ceph-mon[24727]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:57:43.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:42 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:57:43.406 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 05:57:43 smithi139 sudo[39464]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:57:43.407 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 05:57:43 smithi139 sudo[39464]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:43.407 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 05:57:43 smithi139 sudo[39464]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:43.658 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 05:57:43 smithi139 sudo[39470]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:57:43.659 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 05:57:43 smithi139 sudo[39470]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:43.659 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 05:57:43 smithi139 sudo[39470]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:43.659 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:57:43.659 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:57:43.659 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:43.659 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.659 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:43.659 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:43.659 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:43.660 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.660 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.660 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:43.660 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:43.660 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:43.660 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.660 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.660 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:57:43.660 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:57:43.660 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:43.660 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.661 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:43.661 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:43.661 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:43.661 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.661 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.661 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:43.661 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:43.661 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:43.661 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.661 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:57:43.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:57:43.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:44.075 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 05:57:43 smithi139 sudo[39489]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:57:44.075 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 05:57:43 smithi139 sudo[39489]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:44.076 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 05:57:43 smithi139 sudo[39489]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:44.336 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39492]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:57:44.336 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39492]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:44.336 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39492]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:44.336 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39500]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:57:44.336 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39500]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:44.587 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39500]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:44.848 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39529]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:57:44.848 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39529]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:44.848 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39521]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:57:44.849 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39521]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:44.849 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39521]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:45.123 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:44 smithi139 sudo[39529]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:45.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:45 smithi139 ceph-mon[29453]: Detected new or changed devices on smithi139 2024-02-10T05:57:45.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:45 smithi139 ceph-mon[29453]: Adjusting osd_memory_target on smithi139 to 5354M 2024-02-10T05:57:45.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:45 smithi139 ceph-mon[29453]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:57:45.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:45 smithi139 ceph-mon[29453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:57:45.385 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:45 smithi139 sudo[39554]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:57:45.385 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:45 smithi139 sudo[39554]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:45.385 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:45 smithi139 sudo[39554]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:45.386 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:45 smithi139 ceph-mon[24727]: Detected new or changed devices on smithi139 2024-02-10T05:57:45.386 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:45 smithi139 ceph-mon[24727]: Adjusting osd_memory_target on smithi139 to 5354M 2024-02-10T05:57:45.386 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:45 smithi139 ceph-mon[24727]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:57:45.386 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:45 smithi139 ceph-mon[24727]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:57:45.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:45 smithi150 ceph-mon[25582]: Detected new or changed devices on smithi139 2024-02-10T05:57:45.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:45 smithi150 ceph-mon[25582]: Adjusting osd_memory_target on smithi139 to 5354M 2024-02-10T05:57:45.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:45 smithi150 ceph-mon[25582]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:57:45.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:45 smithi150 ceph-mon[25582]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:57:45.643 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:45 smithi139 sudo[39594]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:57:45.644 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:45 smithi139 sudo[39594]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:45.644 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:45 smithi139 sudo[39594]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:46.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:45 smithi139 sudo[39638]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:57:46.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:45 smithi139 sudo[39638]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:46.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:45 smithi139 sudo[39638]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:46.136 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:45 smithi150 sudo[27088]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:57:46.136 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:45 smithi150 sudo[27088]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:46.136 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:45 smithi150 sudo[27088]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:46.455 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:46.455 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:57:46.455 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:57:46.455 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:57:46.455 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:57:46.455 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:57:46.455 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T05:57:46.455 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:57:46.455 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:57:46.455 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:57:46.455 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:57:46.456 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:57:46.457 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:57:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 sudo[27091]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:57:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 sudo[27091]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:57:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 sudo[27091]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:57:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:57:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:57:46.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:57:47.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:47 smithi139 ceph-mon[24727]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:57:47.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:47 smithi139 ceph-mon[24727]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:57:47.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:47.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:47 smithi139 ceph-mon[29453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:57:47.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:47 smithi139 ceph-mon[29453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:57:47.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:47 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:47.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:47 smithi150 ceph-mon[25582]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:57:47.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:47 smithi150 ceph-mon[25582]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:57:47.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:47 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:57:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[24727]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1177231732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "433e8787-cfe9-4309-b185-473bc2d38f35"}]: dispatch 2024-02-10T05:57:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1177231732' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "433e8787-cfe9-4309-b185-473bc2d38f35"}]': finished 2024-02-10T05:57:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[24727]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T05:57:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:57:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1365832908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[29453]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1177231732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "433e8787-cfe9-4309-b185-473bc2d38f35"}]: dispatch 2024-02-10T05:57:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1177231732' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "433e8787-cfe9-4309-b185-473bc2d38f35"}]': finished 2024-02-10T05:57:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[29453]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T05:57:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:57:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:48 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1365832908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:48 smithi150 ceph-mon[25582]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:48 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1177231732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "433e8787-cfe9-4309-b185-473bc2d38f35"}]: dispatch 2024-02-10T05:57:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:48 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1177231732' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "433e8787-cfe9-4309-b185-473bc2d38f35"}]': finished 2024-02-10T05:57:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:48 smithi150 ceph-mon[25582]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T05:57:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:48 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:57:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:48 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1365832908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:49 smithi139 ceph-mon[24727]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:50.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:49 smithi139 ceph-mon[29453]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:50.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:49 smithi150 ceph-mon[25582]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:52.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:52 smithi139 ceph-mon[24727]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:52.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:52 smithi139 ceph-mon[29453]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:52.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:52 smithi150 ceph-mon[25582]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:54.461 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:54 smithi139 ceph-mon[24727]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:54.461 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:54 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:57:54.461 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:54 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:54.461 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:54 smithi139 ceph-mon[29453]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:54.462 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:54 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:57:54.462 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:54 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:54 smithi150 ceph-mon[25582]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:54.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:54 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:57:54.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:54 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:55.326 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:55 smithi139 ceph-mon[24727]: Deploying daemon osd.3 on smithi139 2024-02-10T05:57:55.326 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:55 smithi139 ceph-mon[29453]: Deploying daemon osd.3 on smithi139 2024-02-10T05:57:55.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:55 smithi150 ceph-mon[25582]: Deploying daemon osd.3 on smithi139 2024-02-10T05:57:56.541 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:56 smithi139 ceph-mon[24727]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:56.541 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:56 smithi139 ceph-mon[29453]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:56 smithi150 ceph-mon[25582]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:58.452 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:58 smithi139 ceph-mon[29453]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:58.453 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:58 smithi139 ceph-mon[24727]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:57:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:58 smithi150 ceph-mon[25582]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:00.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:00.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:00.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:00.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:00.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:57:59 smithi139 ceph-mon[24727]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:59 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:59 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:59 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:59 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:00.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:57:59 smithi139 ceph-mon[29453]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:59 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:59 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:59 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:59 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:57:59 smithi150 ceph-mon[25582]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:01.920 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 3 on host 'smithi139' 2024-02-10T05:58:02.449 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:02 smithi139 ceph-mon[24727]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:02.449 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:02 smithi139 ceph-mon[24727]: from='osd.3 [v2:172.21.15.139:6826/684450120,v1:172.21.15.139:6827/684450120]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:58:02.449 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:02 smithi139 ceph-mon[24727]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:58:02.449 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:02 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:02.449 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:02 smithi139 ceph-mon[29453]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:02.449 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:02 smithi139 ceph-mon[29453]: from='osd.3 [v2:172.21.15.139:6826/684450120,v1:172.21.15.139:6827/684450120]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:58:02.449 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:02 smithi139 ceph-mon[29453]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:58:02.449 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:02 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:02.470 DEBUG:teuthology.orchestra.run.smithi139:osd.3> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T05:58:02.472 INFO:tasks.cephadm:Deploying osd.4 on smithi150 with /dev/vg_nvme/lv_4... 2024-02-10T05:58:02.472 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T05:58:02.501 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:02 smithi150 ceph-mon[25582]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:02.501 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:02 smithi150 ceph-mon[25582]: from='osd.3 [v2:172.21.15.139:6826/684450120,v1:172.21.15.139:6827/684450120]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:58:02.501 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:02 smithi150 ceph-mon[25582]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:58:02.502 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:02 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:03.325 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[29453]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:58:03.325 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[29453]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T05:58:03.325 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:03.325 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[29453]: from='osd.3 [v2:172.21.15.139:6826/684450120,v1:172.21.15.139:6827/684450120]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:58:03.325 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[29453]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:58:03.325 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:03.326 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[24727]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:58:03.326 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[24727]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T05:58:03.326 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:03.326 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[24727]: from='osd.3 [v2:172.21.15.139:6826/684450120,v1:172.21.15.139:6827/684450120]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:58:03.326 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[24727]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:58:03.326 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:03.327 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[41172]: 2024-02-10T05:58:03.193+0000 7f17ef73d700 -1 osd.3 0 waiting for initial osdmap 2024-02-10T05:58:03.327 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 05:58:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[41172]: 2024-02-10T05:58:03.197+0000 7f17ec0d8700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:58:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:03 smithi150 ceph-mon[25582]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:58:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:03 smithi150 ceph-mon[25582]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T05:58:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:03 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:03 smithi150 ceph-mon[25582]: from='osd.3 [v2:172.21.15.139:6826/684450120,v1:172.21.15.139:6827/684450120]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:58:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:03 smithi150 ceph-mon[25582]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T05:58:03.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:03 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:04.369 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T05:58:04.369 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T05:58:04.369 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-02-10T05:58:04.369 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-02-10T05:58:04.369 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113563 s, 923 MB/s 2024-02-10T05:58:04.369 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-02-10T05:58:04.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:04 smithi150 ceph-mon[25582]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:04.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:04 smithi150 ceph-mon[25582]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:58:04.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:04 smithi150 ceph-mon[25582]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T05:58:04.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:04 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:04.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:04 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:04.536 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:04 smithi139 ceph-mon[24727]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:04.536 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:04 smithi139 ceph-mon[24727]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:58:04.537 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:04 smithi139 ceph-mon[24727]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T05:58:04.537 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:04 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:04.537 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:04 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:04.537 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:04 smithi139 ceph-mon[29453]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:04.537 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:04 smithi139 ceph-mon[29453]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-02-10T05:58:04.537 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:04 smithi139 ceph-mon[29453]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T05:58:04.537 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:04 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:04.537 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:04 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:04.994 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_4 2024-02-10T05:58:05.347 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:05 smithi150 ceph-mon[25582]: purged_snaps scrub starts 2024-02-10T05:58:05.347 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:05 smithi150 ceph-mon[25582]: purged_snaps scrub ok 2024-02-10T05:58:05.347 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:05 smithi150 ceph-mon[25582]: osd.3 [v2:172.21.15.139:6826/684450120,v1:172.21.15.139:6827/684450120] boot 2024-02-10T05:58:05.347 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:05 smithi150 ceph-mon[25582]: osdmap e25: 4 total, 4 up, 4 in 2024-02-10T05:58:05.347 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:05 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:05.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:05 smithi139 ceph-mon[24727]: purged_snaps scrub starts 2024-02-10T05:58:05.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:05 smithi139 ceph-mon[24727]: purged_snaps scrub ok 2024-02-10T05:58:05.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:05 smithi139 ceph-mon[24727]: osd.3 [v2:172.21.15.139:6826/684450120,v1:172.21.15.139:6827/684450120] boot 2024-02-10T05:58:05.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:05 smithi139 ceph-mon[24727]: osdmap e25: 4 total, 4 up, 4 in 2024-02-10T05:58:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:05 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:05 smithi139 ceph-mon[29453]: purged_snaps scrub starts 2024-02-10T05:58:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:05 smithi139 ceph-mon[29453]: purged_snaps scrub ok 2024-02-10T05:58:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:05 smithi139 ceph-mon[29453]: osd.3 [v2:172.21.15.139:6826/684450120,v1:172.21.15.139:6827/684450120] boot 2024-02-10T05:58:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:05 smithi139 ceph-mon[29453]: osdmap e25: 4 total, 4 up, 4 in 2024-02-10T05:58:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:05 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:06.488 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:06.488 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T05:58:06.489 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:06.489 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.489 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.489 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.489 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.489 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:06.489 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:06.489 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:06.489 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:06.489 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:06.489 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:06 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T05:58:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:06.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:06 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:07.424 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:07 smithi150 ceph-mon[25582]: Detected new or changed devices on smithi139 2024-02-10T05:58:07.425 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:07 smithi150 ceph-mon[25582]: Adjusting osd_memory_target on smithi139 to 4016M 2024-02-10T05:58:07.425 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:07 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:58:07.425 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:07 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:58:07.425 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:07 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:07 smithi139 ceph-mon[24727]: Detected new or changed devices on smithi139 2024-02-10T05:58:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:07 smithi139 ceph-mon[24727]: Adjusting osd_memory_target on smithi139 to 4016M 2024-02-10T05:58:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:58:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:58:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:07 smithi139 ceph-mon[29453]: Detected new or changed devices on smithi139 2024-02-10T05:58:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:07 smithi139 ceph-mon[29453]: Adjusting osd_memory_target on smithi139 to 4016M 2024-02-10T05:58:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:58:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:58:07.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:08.405 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:08 smithi150 ceph-mon[25582]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:58:08.406 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:08 smithi150 ceph-mon[25582]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:08 smithi139 ceph-mon[24727]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:58:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:08 smithi139 ceph-mon[24727]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:08 smithi139 ceph-mon[29453]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:58:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:08 smithi139 ceph-mon[29453]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/766965561' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b23be8-0d14-4bf6-98cc-ee09db996d60"}]: dispatch 2024-02-10T05:58:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b23be8-0d14-4bf6-98cc-ee09db996d60"}]: dispatch 2024-02-10T05:58:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66b23be8-0d14-4bf6-98cc-ee09db996d60"}]': finished 2024-02-10T05:58:10.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[24727]: osdmap e27: 5 total, 4 up, 5 in 2024-02-10T05:58:10.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:10.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[24727]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:10.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/3590536693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:58:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/766965561' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b23be8-0d14-4bf6-98cc-ee09db996d60"}]: dispatch 2024-02-10T05:58:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b23be8-0d14-4bf6-98cc-ee09db996d60"}]: dispatch 2024-02-10T05:58:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66b23be8-0d14-4bf6-98cc-ee09db996d60"}]': finished 2024-02-10T05:58:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[29453]: osdmap e27: 5 total, 4 up, 5 in 2024-02-10T05:58:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[29453]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:09 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/3590536693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:58:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:09 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/766965561' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b23be8-0d14-4bf6-98cc-ee09db996d60"}]: dispatch 2024-02-10T05:58:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:09 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b23be8-0d14-4bf6-98cc-ee09db996d60"}]: dispatch 2024-02-10T05:58:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:09 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66b23be8-0d14-4bf6-98cc-ee09db996d60"}]': finished 2024-02-10T05:58:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:09 smithi150 ceph-mon[25582]: osdmap e27: 5 total, 4 up, 5 in 2024-02-10T05:58:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:09 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:09 smithi150 ceph-mon[25582]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:09 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/3590536693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:58:12.327 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:12 smithi150 ceph-mon[25582]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:12.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:12 smithi139 ceph-mon[24727]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:12.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:12 smithi139 ceph-mon[29453]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:14.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:14 smithi150 ceph-mon[25582]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:14.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:14 smithi139 ceph-mon[24727]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:14.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:14 smithi139 ceph-mon[29453]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:15.310 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:15 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:58:15.310 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:15 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:15.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:15 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:58:15.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:15 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:15.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:15 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:58:15.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:15 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:16.294 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:16 smithi150 ceph-mon[25582]: Deploying daemon osd.4 on smithi150 2024-02-10T05:58:16.294 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:16 smithi150 ceph-mon[25582]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:16 smithi139 ceph-mon[24727]: Deploying daemon osd.4 on smithi150 2024-02-10T05:58:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:16 smithi139 ceph-mon[24727]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:16.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:16 smithi139 ceph-mon[29453]: Deploying daemon osd.4 on smithi150 2024-02-10T05:58:16.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:16 smithi139 ceph-mon[29453]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:18.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:18 smithi150 ceph-mon[25582]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:18 smithi139 ceph-mon[24727]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:18.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:18 smithi139 ceph-mon[29453]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:20.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:19 smithi139 ceph-mon[24727]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:20.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:20.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:20.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:20.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:19 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:20.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:19 smithi139 ceph-mon[29453]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:20.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:20.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:20.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:20.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:19 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:20.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:19 smithi150 ceph-mon[25582]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:20.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:20.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:20.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:20.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:19 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:22.015 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 4 on host 'smithi150' 2024-02-10T05:58:22.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:22 smithi139 ceph-mon[24727]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:22.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:22 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:22.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:22 smithi139 ceph-mon[29453]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:22.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:22 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:22 smithi150 ceph-mon[25582]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:22 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:22.940 DEBUG:teuthology.orchestra.run.smithi150:osd.4> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T05:58:22.942 INFO:tasks.cephadm:Deploying osd.5 on smithi150 with /dev/vg_nvme/lv_3... 2024-02-10T05:58:22.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T05:58:23.509 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:23 smithi150 ceph-mon[25582]: from='osd.4 [v2:172.21.15.150:6800/290935797,v1:172.21.15.150:6801/290935797]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:58:23.509 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:23 smithi150 ceph-mon[25582]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:58:23.509 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:23 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:23.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:23 smithi139 ceph-mon[24727]: from='osd.4 [v2:172.21.15.150:6800/290935797,v1:172.21.15.150:6801/290935797]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:58:23.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:23 smithi139 ceph-mon[24727]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:58:23.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:23 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:23.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:23 smithi139 ceph-mon[29453]: from='osd.4 [v2:172.21.15.150:6800/290935797,v1:172.21.15.150:6801/290935797]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:58:23.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:23 smithi139 ceph-mon[29453]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:58:23.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:23 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:24.481 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:24 smithi150 ceph-mon[25582]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:24.481 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:24 smithi150 ceph-mon[25582]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:58:24.481 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:24 smithi150 ceph-mon[25582]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T05:58:24.481 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:24 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:24.481 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:24 smithi150 ceph-mon[25582]: from='osd.4 [v2:172.21.15.150:6800/290935797,v1:172.21.15.150:6801/290935797]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:24.481 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:24 smithi150 ceph-mon[25582]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:24.482 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 05:58:24 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[29022]: 2024-02-10T05:58:24.206+0000 7f48a7d63700 -1 osd.4 0 waiting for initial osdmap 2024-02-10T05:58:24.482 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 05:58:24 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[29022]: 2024-02-10T05:58:24.211+0000 7f48a26fa700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:58:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[24727]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[24727]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:58:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[24727]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T05:58:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[24727]: from='osd.4 [v2:172.21.15.150:6800/290935797,v1:172.21.15.150:6801/290935797]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[24727]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[29453]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[29453]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:58:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[29453]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T05:58:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[29453]: from='osd.4 [v2:172.21.15.150:6800/290935797,v1:172.21.15.150:6801/290935797]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:24 smithi139 ceph-mon[29453]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:25.470 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T05:58:25.471 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T05:58:25.471 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-02-10T05:58:25.471 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-02-10T05:58:25.471 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127345 s, 823 MB/s 2024-02-10T05:58:25.471 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-02-10T05:58:25.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:25 smithi150 ceph-mon[25582]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:58:25.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:25 smithi150 ceph-mon[25582]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T05:58:25.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:25 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:25.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:25 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:25.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:25 smithi150 ceph-mon[25582]: osd.4 [v2:172.21.15.150:6800/290935797,v1:172.21.15.150:6801/290935797] boot 2024-02-10T05:58:25.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:25 smithi150 ceph-mon[25582]: osdmap e30: 5 total, 5 up, 5 in 2024-02-10T05:58:25.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:25 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:25.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[24727]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:58:25.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[24727]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T05:58:25.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:25.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:25.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[24727]: osd.4 [v2:172.21.15.150:6800/290935797,v1:172.21.15.150:6801/290935797] boot 2024-02-10T05:58:25.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[24727]: osdmap e30: 5 total, 5 up, 5 in 2024-02-10T05:58:25.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:25.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[29453]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:58:25.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[29453]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T05:58:25.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:25.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:25.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[29453]: osd.4 [v2:172.21.15.150:6800/290935797,v1:172.21.15.150:6801/290935797] boot 2024-02-10T05:58:25.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[29453]: osdmap e30: 5 total, 5 up, 5 in 2024-02-10T05:58:25.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:25 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:26.032 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_3 2024-02-10T05:58:26.359 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:26 smithi150 ceph-mon[25582]: purged_snaps scrub starts 2024-02-10T05:58:26.359 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:26 smithi150 ceph-mon[25582]: purged_snaps scrub ok 2024-02-10T05:58:26.359 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:26 smithi150 ceph-mon[25582]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:26.359 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:26 smithi150 ceph-mon[25582]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T05:58:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:26 smithi139 ceph-mon[24727]: purged_snaps scrub starts 2024-02-10T05:58:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:26 smithi139 ceph-mon[24727]: purged_snaps scrub ok 2024-02-10T05:58:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:26 smithi139 ceph-mon[24727]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:26 smithi139 ceph-mon[24727]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T05:58:26.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:26 smithi139 ceph-mon[29453]: purged_snaps scrub starts 2024-02-10T05:58:26.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:26 smithi139 ceph-mon[29453]: purged_snaps scrub ok 2024-02-10T05:58:26.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:26 smithi139 ceph-mon[29453]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:58:26.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:26 smithi139 ceph-mon[29453]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T05:58:28.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:28 smithi150 ceph-mon[25582]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:28.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:28 smithi150 ceph-mon[25582]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T05:58:28.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:28 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:28.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:28 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:28.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:28 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:28.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:28 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:28.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:28 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:28.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:28 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:28.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:28 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:28.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:28 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[24727]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[24727]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T05:58:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:28.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:28.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:28.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:28.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:28.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:28.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:28.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:28.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[29453]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:28.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[29453]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T05:58:28.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:28.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:28.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:28.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:28.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:28.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:28.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:28.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:28 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:29 smithi150 ceph-mon[25582]: Detected new or changed devices on smithi150 2024-02-10T05:58:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:29 smithi150 ceph-mon[25582]: Adjusting osd_memory_target on smithi150 to 17088M 2024-02-10T05:58:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:29 smithi150 ceph-mon[25582]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T05:58:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:58:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:58:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:29.552 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[29453]: Detected new or changed devices on smithi150 2024-02-10T05:58:29.552 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[29453]: Adjusting osd_memory_target on smithi150 to 17088M 2024-02-10T05:58:29.552 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[29453]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T05:58:29.552 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:58:29.552 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:58:29.552 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:29.552 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[24727]: Detected new or changed devices on smithi150 2024-02-10T05:58:29.552 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[24727]: Adjusting osd_memory_target on smithi150 to 17088M 2024-02-10T05:58:29.553 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[24727]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T05:58:29.553 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:58:29.553 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:58:29.553 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:30.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:30 smithi139 ceph-mon[24727]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:58:30.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:30 smithi139 ceph-mon[24727]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:30.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:30 smithi139 ceph-mon[24727]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T05:58:30.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:30 smithi139 ceph-mon[29453]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:58:30.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:30 smithi139 ceph-mon[29453]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:30.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:30 smithi139 ceph-mon[29453]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T05:58:30.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:30 smithi150 ceph-mon[25582]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:58:30.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:30 smithi150 ceph-mon[25582]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:30.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:30 smithi150 ceph-mon[25582]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T05:58:31.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/3816976408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ffbdbb7-af3d-4023-960e-a61b22010132"}]: dispatch 2024-02-10T05:58:31.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ffbdbb7-af3d-4023-960e-a61b22010132"}]: dispatch 2024-02-10T05:58:31.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ffbdbb7-af3d-4023-960e-a61b22010132"}]': finished 2024-02-10T05:58:31.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[24727]: osdmap e35: 6 total, 5 up, 6 in 2024-02-10T05:58:31.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:31.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:58:31.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:58:31.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/3816976408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ffbdbb7-af3d-4023-960e-a61b22010132"}]: dispatch 2024-02-10T05:58:31.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ffbdbb7-af3d-4023-960e-a61b22010132"}]: dispatch 2024-02-10T05:58:31.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ffbdbb7-af3d-4023-960e-a61b22010132"}]': finished 2024-02-10T05:58:31.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[29453]: osdmap e35: 6 total, 5 up, 6 in 2024-02-10T05:58:31.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:31.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:58:31.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:58:31.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:31 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/3816976408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ffbdbb7-af3d-4023-960e-a61b22010132"}]: dispatch 2024-02-10T05:58:31.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:31 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ffbdbb7-af3d-4023-960e-a61b22010132"}]: dispatch 2024-02-10T05:58:31.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:31 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ffbdbb7-af3d-4023-960e-a61b22010132"}]': finished 2024-02-10T05:58:31.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:31 smithi150 ceph-mon[25582]: osdmap e35: 6 total, 5 up, 6 in 2024-02-10T05:58:31.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:31.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:58:31.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:58:32.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:32 smithi139 ceph-mon[24727]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:32.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:32 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/160818605' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:58:32.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:32 smithi139 ceph-mon[29453]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:32.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:32 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/160818605' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:58:32.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:32 smithi150 ceph-mon[25582]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:32.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:32 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/160818605' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:58:34.548 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:34 smithi150 ceph-mon[25582]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-10T05:58:34.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:34 smithi139 ceph-mon[24727]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-10T05:58:34.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:34 smithi139 ceph-mon[29453]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-10T05:58:36.395 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:36 smithi150 ceph-mon[25582]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:58:36.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:36 smithi139 ceph-mon[24727]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:58:36.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:36 smithi139 ceph-mon[29453]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:58:37.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:37 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:58:37.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:37 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:37.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:37 smithi150 ceph-mon[25582]: Deploying daemon osd.5 on smithi150 2024-02-10T05:58:37.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:37 smithi150 ceph-mon[25582]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:58:37.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:37 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:58:37.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:37 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:37.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:37 smithi139 ceph-mon[24727]: Deploying daemon osd.5 on smithi150 2024-02-10T05:58:37.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:37 smithi139 ceph-mon[24727]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:58:37.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:37 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:58:37.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:37 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:37.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:37 smithi139 ceph-mon[29453]: Deploying daemon osd.5 on smithi150 2024-02-10T05:58:37.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:37 smithi139 ceph-mon[29453]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:58:40.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:39 smithi150 ceph-mon[25582]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T05:58:40.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:39 smithi139 ceph-mon[24727]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T05:58:40.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:39 smithi139 ceph-mon[29453]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T05:58:42.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:42 smithi150 ceph-mon[25582]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-02-10T05:58:42.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:42 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:42.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:42 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:42.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:42 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:42.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:42 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:42 smithi139 ceph-mon[24727]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-02-10T05:58:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:42 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:42 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:42.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:42 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:42.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:42 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:42 smithi139 ceph-mon[29453]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-02-10T05:58:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:42 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:42 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:42 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:42 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:44.347 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 5 on host 'smithi150' 2024-02-10T05:58:44.362 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:44 smithi150 ceph-mon[25582]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-10T05:58:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:44 smithi139 ceph-mon[24727]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-10T05:58:44.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:44 smithi139 ceph-mon[29453]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-10T05:58:44.965 DEBUG:teuthology.orchestra.run.smithi150:osd.5> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T05:58:44.968 INFO:tasks.cephadm:Deploying osd.6 on smithi150 with /dev/vg_nvme/lv_2... 2024-02-10T05:58:44.968 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T05:58:45.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:45 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:45.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:45 smithi150 ceph-mon[25582]: from='osd.5 [v2:172.21.15.150:6808/1581167176,v1:172.21.15.150:6809/1581167176]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:58:45.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:45 smithi150 ceph-mon[25582]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:58:45.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:45 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:45.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:45 smithi150 ceph-mon[25582]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:45.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:45 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:45.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:45 smithi139 ceph-mon[24727]: from='osd.5 [v2:172.21.15.150:6808/1581167176,v1:172.21.15.150:6809/1581167176]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:58:45.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:45 smithi139 ceph-mon[24727]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:58:45.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:45 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:45.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:45 smithi139 ceph-mon[24727]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:45.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:45 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:45.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:45 smithi139 ceph-mon[29453]: from='osd.5 [v2:172.21.15.150:6808/1581167176,v1:172.21.15.150:6809/1581167176]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:58:45.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:45 smithi139 ceph-mon[29453]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:58:45.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:45 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:45.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:45 smithi139 ceph-mon[29453]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:46.551 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:46 smithi150 ceph-mon[25582]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:58:46.551 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:46 smithi150 ceph-mon[25582]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T05:58:46.551 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:46 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:46.551 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:46 smithi150 ceph-mon[25582]: from='osd.5 [v2:172.21.15.150:6808/1581167176,v1:172.21.15.150:6809/1581167176]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:46.551 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:46 smithi150 ceph-mon[25582]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:46.552 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 05:58:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[31899]: 2024-02-10T05:58:46.362+0000 7f164720d700 -1 osd.5 0 waiting for initial osdmap 2024-02-10T05:58:46.552 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 05:58:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[31899]: 2024-02-10T05:58:46.367+0000 7f1642ba6700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:58:46.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:46 smithi139 ceph-mon[24727]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:58:46.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:46 smithi139 ceph-mon[24727]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T05:58:46.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:46 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:46.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:46 smithi139 ceph-mon[24727]: from='osd.5 [v2:172.21.15.150:6808/1581167176,v1:172.21.15.150:6809/1581167176]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:46.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:46 smithi139 ceph-mon[24727]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:46 smithi139 ceph-mon[29453]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:58:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:46 smithi139 ceph-mon[29453]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T05:58:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:46 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:46 smithi139 ceph-mon[29453]: from='osd.5 [v2:172.21.15.150:6808/1581167176,v1:172.21.15.150:6809/1581167176]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:46 smithi139 ceph-mon[29453]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:58:47.378 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T05:58:47.379 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T05:58:47.379 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-02-10T05:58:47.379 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-02-10T05:58:47.379 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125241 s, 837 MB/s 2024-02-10T05:58:47.379 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-02-10T05:58:47.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:47 smithi150 ceph-mon[25582]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:58:47.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:47 smithi150 ceph-mon[25582]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T05:58:47.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:47 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:47.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:47 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:47.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:47 smithi150 ceph-mon[25582]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:47.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:47 smithi139 ceph-mon[24727]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:58:47.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:47 smithi139 ceph-mon[24727]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T05:58:47.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:47.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:47 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:47.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:47 smithi139 ceph-mon[24727]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:47 smithi139 ceph-mon[29453]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:58:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:47 smithi139 ceph-mon[29453]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T05:58:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:47 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:47 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:47 smithi139 ceph-mon[29453]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:58:48.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_2 2024-02-10T05:58:48.507 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:48 smithi150 ceph-mon[25582]: purged_snaps scrub starts 2024-02-10T05:58:48.507 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:48 smithi150 ceph-mon[25582]: purged_snaps scrub ok 2024-02-10T05:58:48.507 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:48 smithi150 ceph-mon[25582]: osd.5 [v2:172.21.15.150:6808/1581167176,v1:172.21.15.150:6809/1581167176] boot 2024-02-10T05:58:48.507 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:48 smithi150 ceph-mon[25582]: osdmap e38: 6 total, 6 up, 6 in 2024-02-10T05:58:48.507 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:48 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:48.508 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:48 smithi150 ceph-mon[25582]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T05:58:48.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[24727]: purged_snaps scrub starts 2024-02-10T05:58:48.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[24727]: purged_snaps scrub ok 2024-02-10T05:58:48.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[24727]: osd.5 [v2:172.21.15.150:6808/1581167176,v1:172.21.15.150:6809/1581167176] boot 2024-02-10T05:58:48.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[24727]: osdmap e38: 6 total, 6 up, 6 in 2024-02-10T05:58:48.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:48.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[24727]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T05:58:48.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[29453]: purged_snaps scrub starts 2024-02-10T05:58:48.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[29453]: purged_snaps scrub ok 2024-02-10T05:58:48.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[29453]: osd.5 [v2:172.21.15.150:6808/1581167176,v1:172.21.15.150:6809/1581167176] boot 2024-02-10T05:58:48.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[29453]: osdmap e38: 6 total, 6 up, 6 in 2024-02-10T05:58:48.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:48.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:48 smithi139 ceph-mon[29453]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T05:58:49.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:49 smithi150 ceph-mon[25582]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:58:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:49 smithi150 ceph-mon[25582]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T05:58:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:49 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:49 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:49 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:49 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:49 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:49 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:49 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:49 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:49 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[24727]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:58:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[24727]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T05:58:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[29453]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[29453]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:50.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:50.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:50.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:50.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:49 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:58:50.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:50 smithi150 ceph-mon[25582]: Detected new or changed devices on smithi150 2024-02-10T05:58:50.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:50 smithi150 ceph-mon[25582]: Adjusting osd_memory_target on smithi150 to 8544M 2024-02-10T05:58:50.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:50 smithi150 ceph-mon[25582]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:58:50.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:50 smithi150 ceph-mon[25582]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T05:58:50.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:50 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:58:50.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:50 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:58:50.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:50 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:51.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[29453]: Detected new or changed devices on smithi150 2024-02-10T05:58:51.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[29453]: Adjusting osd_memory_target on smithi150 to 8544M 2024-02-10T05:58:51.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[29453]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:58:51.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[29453]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T05:58:51.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:58:51.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:58:51.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:51.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[24727]: Detected new or changed devices on smithi150 2024-02-10T05:58:51.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[24727]: Adjusting osd_memory_target on smithi150 to 8544M 2024-02-10T05:58:51.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[24727]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:58:51.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[24727]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T05:58:51.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:58:51.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:58:51.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:50 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:51.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:51 smithi150 ceph-mon[25582]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:58:51.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:51 smithi150 ceph-mon[25582]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:58:51.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:51 smithi150 ceph-mon[25582]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T05:58:52.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:51 smithi139 ceph-mon[24727]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:58:52.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:51 smithi139 ceph-mon[24727]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:58:52.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:51 smithi139 ceph-mon[24727]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T05:58:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:51 smithi139 ceph-mon[29453]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:58:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:51 smithi139 ceph-mon[29453]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:58:52.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:51 smithi139 ceph-mon[29453]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T05:58:53.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:52 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/403945491' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa1bab2a-38b3-429a-ba5a-ff7499fe9b74"}]: dispatch 2024-02-10T05:58:53.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:52 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa1bab2a-38b3-429a-ba5a-ff7499fe9b74"}]: dispatch 2024-02-10T05:58:53.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:52 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa1bab2a-38b3-429a-ba5a-ff7499fe9b74"}]': finished 2024-02-10T05:58:53.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:52 smithi139 ceph-mon[24727]: osdmap e43: 7 total, 6 up, 7 in 2024-02-10T05:58:53.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:52 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:58:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:52 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/403945491' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa1bab2a-38b3-429a-ba5a-ff7499fe9b74"}]: dispatch 2024-02-10T05:58:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:52 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa1bab2a-38b3-429a-ba5a-ff7499fe9b74"}]: dispatch 2024-02-10T05:58:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:52 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa1bab2a-38b3-429a-ba5a-ff7499fe9b74"}]': finished 2024-02-10T05:58:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:52 smithi139 ceph-mon[29453]: osdmap e43: 7 total, 6 up, 7 in 2024-02-10T05:58:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:52 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:58:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:52 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/403945491' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa1bab2a-38b3-429a-ba5a-ff7499fe9b74"}]: dispatch 2024-02-10T05:58:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:52 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa1bab2a-38b3-429a-ba5a-ff7499fe9b74"}]: dispatch 2024-02-10T05:58:53.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:52 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa1bab2a-38b3-429a-ba5a-ff7499fe9b74"}]': finished 2024-02-10T05:58:53.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:52 smithi150 ceph-mon[25582]: osdmap e43: 7 total, 6 up, 7 in 2024-02-10T05:58:53.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:52 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:58:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:53 smithi139 ceph-mon[24727]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-10T05:58:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:53 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/3728026305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:58:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:53 smithi139 ceph-mon[29453]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-10T05:58:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:53 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/3728026305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:58:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:53 smithi150 ceph-mon[25582]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-10T05:58:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:53 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/3728026305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:58:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:54 smithi139 ceph-mon[24727]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:58:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:54 smithi139 ceph-mon[24727]: Cluster is now healthy 2024-02-10T05:58:55.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:54 smithi139 ceph-mon[29453]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:58:55.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:54 smithi139 ceph-mon[29453]: Cluster is now healthy 2024-02-10T05:58:55.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:54 smithi150 ceph-mon[25582]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:58:55.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:54 smithi150 ceph-mon[25582]: Cluster is now healthy 2024-02-10T05:58:55.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:55 smithi150 ceph-mon[25582]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-10T05:58:56.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:55 smithi139 ceph-mon[24727]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-10T05:58:56.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:55 smithi139 ceph-mon[29453]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-10T05:58:58.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:58 smithi139 ceph-mon[24727]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-02-10T05:58:58.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:58 smithi139 ceph-mon[29453]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-02-10T05:58:58.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:58 smithi150 ceph-mon[25582]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-02-10T05:59:00.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:59:00.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:59 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:00.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:59 smithi139 ceph-mon[24727]: Deploying daemon osd.6 on smithi150 2024-02-10T05:59:00.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:58:59 smithi139 ceph-mon[24727]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:59:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:59 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:59:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:59 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:59 smithi139 ceph-mon[29453]: Deploying daemon osd.6 on smithi150 2024-02-10T05:59:00.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:58:59 smithi139 ceph-mon[29453]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:59:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:59 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:59:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:59 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:59 smithi150 ceph-mon[25582]: Deploying daemon osd.6 on smithi150 2024-02-10T05:59:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:58:59 smithi150 ceph-mon[25582]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:59:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:02 smithi150 ceph-mon[25582]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-02-10T05:59:02.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:02 smithi139 ceph-mon[24727]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-02-10T05:59:02.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:02 smithi139 ceph-mon[29453]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-02-10T05:59:04.309 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:04 smithi150 ceph-mon[25582]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:04.309 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:04 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:04.309 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:04 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:04.309 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:04 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:04.309 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:04 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:04.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:04 smithi139 ceph-mon[29453]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:04.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:04 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:04.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:04 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:04.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:04 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:04.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:04 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:04 smithi139 ceph-mon[24727]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:04 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:04 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:04 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:04 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:06.292 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:06 smithi150 ceph-mon[25582]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:06.303 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 6 on host 'smithi150' 2024-02-10T05:59:06.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:06 smithi139 ceph-mon[29453]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:06 smithi139 ceph-mon[24727]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:07.208 DEBUG:teuthology.orchestra.run.smithi150:osd.6> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.6.service 2024-02-10T05:59:07.211 INFO:tasks.cephadm:Deploying osd.7 on smithi150 with /dev/vg_nvme/lv_1... 2024-02-10T05:59:07.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T05:59:07.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:07 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:07.511 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:07 smithi150 ceph-mon[25582]: from='osd.6 [v2:172.21.15.150:6816/382604718,v1:172.21.15.150:6817/382604718]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:59:07.511 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:07 smithi150 ceph-mon[25582]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:59:07.511 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:07 smithi150 ceph-mon[25582]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:07 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:07 smithi139 ceph-mon[24727]: from='osd.6 [v2:172.21.15.150:6816/382604718,v1:172.21.15.150:6817/382604718]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:59:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:07 smithi139 ceph-mon[24727]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:59:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:07 smithi139 ceph-mon[24727]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:07.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:07 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:07.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:07 smithi139 ceph-mon[29453]: from='osd.6 [v2:172.21.15.150:6816/382604718,v1:172.21.15.150:6817/382604718]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:59:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:07 smithi139 ceph-mon[29453]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:59:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:07 smithi139 ceph-mon[29453]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:08.537 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:08 smithi150 ceph-mon[25582]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:59:08.537 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:08 smithi150 ceph-mon[25582]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T05:59:08.537 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:08 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:08.537 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:08 smithi150 ceph-mon[25582]: from='osd.6 [v2:172.21.15.150:6816/382604718,v1:172.21.15.150:6817/382604718]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:08.537 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:08 smithi150 ceph-mon[25582]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:08.537 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:08 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:08.538 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 05:59:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[34777]: 2024-02-10T05:59:08.317+0000 7fb223072700 -1 osd.6 0 waiting for initial osdmap 2024-02-10T05:59:08.538 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 05:59:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[34777]: 2024-02-10T05:59:08.324+0000 7fb21ba05700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:59:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[29453]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:59:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[29453]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T05:59:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[29453]: from='osd.6 [v2:172.21.15.150:6816/382604718,v1:172.21.15.150:6817/382604718]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[29453]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[24727]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:59:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[24727]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T05:59:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[24727]: from='osd.6 [v2:172.21.15.150:6816/382604718,v1:172.21.15.150:6817/382604718]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[24727]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:08 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:09.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:09 smithi139 ceph-mon[24727]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:59:09.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:09 smithi139 ceph-mon[24727]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T05:59:09.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:09 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:09.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:09 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:09.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:09 smithi139 ceph-mon[24727]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:09.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:09 smithi139 ceph-mon[29453]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:59:09.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:09 smithi139 ceph-mon[29453]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T05:59:09.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:09 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:09.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:09 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:09.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:09 smithi139 ceph-mon[29453]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:09.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:09 smithi150 ceph-mon[25582]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:59:09.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:09 smithi150 ceph-mon[25582]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T05:59:09.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:09 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:09.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:09 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:09.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:09 smithi150 ceph-mon[25582]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:59:09.618 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T05:59:09.618 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T05:59:09.618 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-02-10T05:59:09.619 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-02-10T05:59:09.619 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121632 s, 862 MB/s 2024-02-10T05:59:09.619 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-02-10T05:59:10.351 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:10 smithi150 ceph-mon[25582]: purged_snaps scrub starts 2024-02-10T05:59:10.351 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:10 smithi150 ceph-mon[25582]: purged_snaps scrub ok 2024-02-10T05:59:10.351 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:10 smithi150 ceph-mon[25582]: osd.6 [v2:172.21.15.150:6816/382604718,v1:172.21.15.150:6817/382604718] boot 2024-02-10T05:59:10.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:10 smithi150 ceph-mon[25582]: osdmap e46: 7 total, 7 up, 7 in 2024-02-10T05:59:10.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:10 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:10.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:10 smithi150 ceph-mon[25582]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T05:59:10.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_1 2024-02-10T05:59:10.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[24727]: purged_snaps scrub starts 2024-02-10T05:59:10.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[24727]: purged_snaps scrub ok 2024-02-10T05:59:10.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[24727]: osd.6 [v2:172.21.15.150:6816/382604718,v1:172.21.15.150:6817/382604718] boot 2024-02-10T05:59:10.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[24727]: osdmap e46: 7 total, 7 up, 7 in 2024-02-10T05:59:10.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:10.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[24727]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T05:59:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[29453]: purged_snaps scrub starts 2024-02-10T05:59:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[29453]: purged_snaps scrub ok 2024-02-10T05:59:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[29453]: osd.6 [v2:172.21.15.150:6816/382604718,v1:172.21.15.150:6817/382604718] boot 2024-02-10T05:59:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[29453]: osdmap e46: 7 total, 7 up, 7 in 2024-02-10T05:59:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:10 smithi139 ceph-mon[29453]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T05:59:11.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:11 smithi139 ceph-mon[24727]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:11.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:11 smithi139 ceph-mon[29453]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:11.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:11 smithi150 ceph-mon[25582]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:12 smithi150 ceph-mon[25582]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T05:59:12.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:12 smithi139 ceph-mon[24727]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T05:59:12.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:12 smithi139 ceph-mon[29453]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T05:59:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T05:59:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: Detected new or changed devices on smithi150 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: Adjusting osd_memory_target on smithi150 to 5696M 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:13 smithi150 ceph-mon[25582]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:13.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T05:59:13.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:13.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: Detected new or changed devices on smithi150 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: Adjusting osd_memory_target on smithi150 to 5696M 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:13.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:13.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[24727]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:13.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T05:59:13.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:13.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:59:13.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:59:13.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:13.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: Detected new or changed devices on smithi150 2024-02-10T05:59:13.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:13.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:13.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:13.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: Adjusting osd_memory_target on smithi150 to 5696M 2024-02-10T05:59:13.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:13.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:13.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:13.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:13.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:13 smithi139 ceph-mon[29453]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:14.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:14 smithi150 ceph-mon[25582]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T05:59:14.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:14 smithi139 ceph-mon[24727]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T05:59:14.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:14 smithi139 ceph-mon[29453]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T05:59:15.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:15 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/569886405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "309c3600-54dd-4102-b5a6-2080aba9b681"}]: dispatch 2024-02-10T05:59:15.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:15 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "309c3600-54dd-4102-b5a6-2080aba9b681"}]: dispatch 2024-02-10T05:59:15.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:15 smithi150 ceph-mon[25582]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "309c3600-54dd-4102-b5a6-2080aba9b681"}]': finished 2024-02-10T05:59:15.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:15 smithi150 ceph-mon[25582]: osdmap e51: 8 total, 7 up, 8 in 2024-02-10T05:59:15.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:15 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:15.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:15 smithi150 ceph-mon[25582]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:15.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/569886405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "309c3600-54dd-4102-b5a6-2080aba9b681"}]: dispatch 2024-02-10T05:59:15.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "309c3600-54dd-4102-b5a6-2080aba9b681"}]: dispatch 2024-02-10T05:59:15.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[24727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "309c3600-54dd-4102-b5a6-2080aba9b681"}]': finished 2024-02-10T05:59:15.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[24727]: osdmap e51: 8 total, 7 up, 8 in 2024-02-10T05:59:15.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[24727]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:15.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/569886405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "309c3600-54dd-4102-b5a6-2080aba9b681"}]: dispatch 2024-02-10T05:59:15.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "309c3600-54dd-4102-b5a6-2080aba9b681"}]: dispatch 2024-02-10T05:59:15.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[29453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "309c3600-54dd-4102-b5a6-2080aba9b681"}]': finished 2024-02-10T05:59:15.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[29453]: osdmap e51: 8 total, 7 up, 8 in 2024-02-10T05:59:15.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:15.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:15 smithi139 ceph-mon[29453]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:16 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/4105314991' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:59:16.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:16 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/4105314991' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:59:16.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:16 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/4105314991' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:59:17.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:17 smithi150 ceph-mon[25582]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-10T05:59:17.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:17 smithi139 ceph-mon[24727]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-10T05:59:17.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:17 smithi139 ceph-mon[29453]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-10T05:59:19.996 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:19 smithi150 ceph-mon[25582]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-02-10T05:59:20.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:19 smithi139 ceph-mon[24727]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-02-10T05:59:20.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:19 smithi139 ceph-mon[29453]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-02-10T05:59:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:21 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:59:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:21 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:21 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:59:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:21 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:21 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:59:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:21 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:22.292 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:22 smithi150 ceph-mon[25582]: Deploying daemon osd.7 on smithi150 2024-02-10T05:59:22.292 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:22 smithi150 ceph-mon[25582]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:59:22.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:22 smithi139 ceph-mon[29453]: Deploying daemon osd.7 on smithi150 2024-02-10T05:59:22.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:22 smithi139 ceph-mon[29453]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:59:22.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:22 smithi139 ceph-mon[24727]: Deploying daemon osd.7 on smithi150 2024-02-10T05:59:22.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:22 smithi139 ceph-mon[24727]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:59:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:24 smithi139 ceph-mon[24727]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-02-10T05:59:24.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:24 smithi139 ceph-mon[29453]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-02-10T05:59:24.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:24 smithi150 ceph-mon[25582]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-02-10T05:59:26.520 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:26 smithi150 ceph-mon[25582]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-02-10T05:59:26.520 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:26.520 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:26.520 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:26.520 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:26 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:26 smithi139 ceph-mon[24727]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-02-10T05:59:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:26 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:26.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:26 smithi139 ceph-mon[29453]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-02-10T05:59:26.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:26.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:26.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:26.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:26 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:28.525 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:28 smithi150 ceph-mon[25582]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-10T05:59:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:28 smithi139 ceph-mon[24727]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-10T05:59:28.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:28 smithi139 ceph-mon[29453]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-10T05:59:28.811 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 7 on host 'smithi150' 2024-02-10T05:59:29.394 DEBUG:teuthology.orchestra.run.smithi150:osd.7> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.7.service 2024-02-10T05:59:29.396 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-10T05:59:29.396 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T05:59:29.962 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:29.962 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:29 smithi139 ceph-mon[24727]: from='osd.7 [v2:172.21.15.150:6824/2923308500,v1:172.21.15.150:6825/2923308500]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:59:29.962 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:29 smithi139 ceph-mon[24727]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:59:29.962 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:29 smithi139 ceph-mon[24727]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:29.962 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:29 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:29.963 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:29.963 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:29 smithi139 ceph-mon[29453]: from='osd.7 [v2:172.21.15.150:6824/2923308500,v1:172.21.15.150:6825/2923308500]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:59:29.963 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:29 smithi139 ceph-mon[29453]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:59:29.963 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:29 smithi139 ceph-mon[29453]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:29.963 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:29 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:30.067 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:30.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:29 smithi150 ceph-mon[25582]: from='osd.7 [v2:172.21.15.150:6824/2923308500,v1:172.21.15.150:6825/2923308500]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:59:30.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:29 smithi150 ceph-mon[25582]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:59:30.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:29 smithi150 ceph-mon[25582]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:30.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:29 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:31.038 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:30 smithi139 ceph-mon[24727]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:59:31.039 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:30 smithi139 ceph-mon[24727]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T05:59:31.039 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:30 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:31.039 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:30 smithi139 ceph-mon[24727]: from='osd.7 [v2:172.21.15.150:6824/2923308500,v1:172.21.15.150:6825/2923308500]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:31.039 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:30 smithi139 ceph-mon[24727]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:31.039 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:30 smithi139 ceph-mon[29453]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:59:31.039 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:30 smithi139 ceph-mon[29453]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T05:59:31.039 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:30 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:31.039 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:30 smithi139 ceph-mon[29453]: from='osd.7 [v2:172.21.15.150:6824/2923308500,v1:172.21.15.150:6825/2923308500]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:31.039 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:30 smithi139 ceph-mon[29453]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:31.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:30 smithi150 ceph-mon[25582]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:59:31.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:30 smithi150 ceph-mon[25582]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T05:59:31.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:30 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:31.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:30 smithi150 ceph-mon[25582]: from='osd.7 [v2:172.21.15.150:6824/2923308500,v1:172.21.15.150:6825/2923308500]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:31.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:30 smithi150 ceph-mon[25582]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T05:59:31.091 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 05:59:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[37768]: 2024-02-10T05:59:30.822+0000 7f14b6c5b700 -1 osd.7 0 waiting for initial osdmap 2024-02-10T05:59:31.091 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 05:59:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[37768]: 2024-02-10T05:59:30.829+0000 7f14b1df3700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:59:31.443 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T05:59:32.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[29453]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:59:32.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[29453]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T05:59:32.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:32.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:32.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[29453]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:32.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:59:32.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:59:32.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/524674140' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:59:32.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[24727]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:59:32.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[24727]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T05:59:32.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:32.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:32.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[24727]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:32.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:59:32.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:59:32.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:31 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/524674140' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:59:32.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:31 smithi150 ceph-mon[25582]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-10T05:59:32.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:31 smithi150 ceph-mon[25582]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T05:59:32.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:32.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:32.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:31 smithi150 ceph-mon[25582]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:59:32.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:59:32.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:31 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:59:32.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:31 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/524674140' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:59:32.127 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1707544749,"num_in_osds":8,"osd_in_since":1707544754,"num_remapped_pgs":0} 2024-02-10T05:59:33.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: purged_snaps scrub starts 2024-02-10T05:59:33.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: purged_snaps scrub ok 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: osd.7 [v2:172.21.15.150:6824/2923308500,v1:172.21.15.150:6825/2923308500] boot 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: osdmap e54: 8 total, 8 up, 8 in 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:33.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: purged_snaps scrub starts 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: purged_snaps scrub ok 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: osd.7 [v2:172.21.15.150:6824/2923308500,v1:172.21.15.150:6825/2923308500] boot 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: osdmap e54: 8 total, 8 up, 8 in 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:33.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:33.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:33.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:32 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: purged_snaps scrub starts 2024-02-10T05:59:33.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: purged_snaps scrub ok 2024-02-10T05:59:33.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:33.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: osd.7 [v2:172.21.15.150:6824/2923308500,v1:172.21.15.150:6825/2923308500] boot 2024-02-10T05:59:33.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: osdmap e54: 8 total, 8 up, 8 in 2024-02-10T05:59:33.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:33.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:59:33.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:33.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:33.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:33.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:32 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:33.128 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T05:59:34.063 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:33 smithi139 ceph-mon[24727]: Detected new or changed devices on smithi150 2024-02-10T05:59:34.063 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:33 smithi139 ceph-mon[24727]: Adjusting osd_memory_target on smithi150 to 4272M 2024-02-10T05:59:34.063 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:33 smithi139 ceph-mon[24727]: osdmap e55: 8 total, 8 up, 8 in 2024-02-10T05:59:34.063 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:33 smithi139 ceph-mon[24727]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:34.064 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:33 smithi139 ceph-mon[29453]: Detected new or changed devices on smithi150 2024-02-10T05:59:34.064 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:33 smithi139 ceph-mon[29453]: Adjusting osd_memory_target on smithi150 to 4272M 2024-02-10T05:59:34.064 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:33 smithi139 ceph-mon[29453]: osdmap e55: 8 total, 8 up, 8 in 2024-02-10T05:59:34.064 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:33 smithi139 ceph-mon[29453]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:34.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:33 smithi150 ceph-mon[25582]: Detected new or changed devices on smithi150 2024-02-10T05:59:34.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:33 smithi150 ceph-mon[25582]: Adjusting osd_memory_target on smithi150 to 4272M 2024-02-10T05:59:34.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:33 smithi150 ceph-mon[25582]: osdmap e55: 8 total, 8 up, 8 in 2024-02-10T05:59:34.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:33 smithi150 ceph-mon[25582]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:35.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:34 smithi139 ceph-mon[24727]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T05:59:35.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:34 smithi139 ceph-mon[29453]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T05:59:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T05:59:35.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:34 smithi150 ceph-mon[25582]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T05:59:35.862 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:35 smithi139 ceph-mon[24727]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T05:59:35.862 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:35 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/298529928' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:59:35.862 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:35 smithi139 ceph-mon[24727]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:35.862 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:35 smithi139 ceph-mon[29453]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T05:59:35.862 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:35 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/298529928' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:59:35.862 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:35 smithi139 ceph-mon[29453]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:35.885 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1707544771,"num_in_osds":8,"osd_in_since":1707544754,"num_remapped_pgs":0} 2024-02-10T05:59:35.885 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:59:36.097 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:59:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:35 smithi150 ceph-mon[25582]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T05:59:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:35 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/298529928' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:59:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:35 smithi150 ceph-mon[25582]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:37.118 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:36 smithi139 ceph-mon[29453]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T05:59:37.118 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:36 smithi139 ceph-mon[24727]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T05:59:37.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:36 smithi150 ceph-mon[25582]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T05:59:37.795 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T05:59:37.795 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":58,"fsid":"ce44e742-c7d8-11ee-95b8-87774f69a715","created":"2024-02-10T05:54:37.242368+0000","modified":"2024-02-10T05:59:35.853064+0000","last_up_change":"2024-02-10T05:59:31.819060+0000","last_in_change":"2024-02-10T05:59:14.928470+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:57:41.413330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6803","nonce":3875590395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6805","nonce":3875590395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6809","nonce":3875590395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6807","nonce":3875590395}]},"public_addr":"172.21.15.139:6803/3875590395","cluster_addr":"172.21.15.139:6805/3875590395","heartbeat_back_addr":"172.21.15.139:6809/3875590395","heartbeat_front_addr":"172.21.15.139:6807/3875590395","state":["exists","up"]},{"osd":1,"uuid":"a39bbeb2-0fd3-4584-bde1-2578c452924e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6811","nonce":2796500287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6813","nonce":2796500287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6817","nonce":2796500287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6815","nonce":2796500287}]},"public_addr":"172.21.15.139:6811/2796500287","cluster_addr":"172.21.15.139:6813/2796500287","heartbeat_back_addr":"172.21.15.139:6817/2796500287","heartbeat_front_addr":"172.21.15.139:6815/2796500287","state":["exists","up"]},{"osd":2,"uuid":"1f801d19-78c0-40e6-8f36-efd49380be35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6819","nonce":872620670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6821","nonce":872620670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6825","nonce":872620670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6823","nonce":872620670}]},"public_addr":"172.21.15.139:6819/872620670","cluster_addr":"172.21.15.139:6821/872620670","heartbeat_back_addr":"172.21.15.139:6825/872620670","heartbeat_front_addr":"172.21.15.139:6823/872620670","state":["exists","up"]},{"osd":3,"uuid":"433e8787-cfe9-4309-b185-473bc2d38f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6827","nonce":684450120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6829","nonce":684450120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6833","nonce":684450120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6831","nonce":684450120}]},"public_addr":"172.21.15.139:6827/684450120","cluster_addr":"172.21.15.139:6829/684450120","heartbeat_back_addr":"172.21.15.139:6833/684450120","heartbeat_front_addr":"172.21.15.139:6831/684450120","state":["exists","up"]},{"osd":4,"uuid":"66b23be8-0d14-4bf6-98cc-ee09db996d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6801","nonce":290935797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6803","nonce":290935797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6807","nonce":290935797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6805","nonce":290935797}]},"public_addr":"172.21.15.150:6801/290935797","cluster_addr":"172.21.15.150:6803/290935797","heartbeat_back_addr":"172.21.15.150:6807/290935797","heartbeat_front_addr":"172.21.15.150:6805/290935797","state":["exists","up"]},{"osd":5,"uuid":"0ffbdbb7-af3d-4023-960e-a61b22010132","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6809","nonce":1581167176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6811","nonce":1581167176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6815","nonce":1581167176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6813","nonce":1581167176}]},"public_addr":"172.21.15.150:6809/1581167176","cluster_addr":"172.21.15.150:6811/1581167176","heartbeat_back_addr":"172.21.15.150:6815/1581167176","heartbeat_front_addr":"172.21.15.150:6813/1581167176","state":["exists","up"]},{"osd":6,"uuid":"aa1bab2a-38b3-429a-ba5a-ff7499fe9b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6817","nonce":382604718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6819","nonce":382604718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6823","nonce":382604718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6821","nonce":382604718}]},"public_addr":"172.21.15.150:6817/382604718","cluster_addr":"172.21.15.150:6819/382604718","heartbeat_back_addr":"172.21.15.150:6823/382604718","heartbeat_front_addr":"172.21.15.150:6821/382604718","state":["exists","up"]},{"osd":7,"uuid":"309c3600-54dd-4102-b5a6-2080aba9b681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6825","nonce":2923308500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6827","nonce":2923308500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6831","nonce":2923308500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6829","nonce":2923308500}]},"public_addr":"172.21.15.150:6825/2923308500","cluster_addr":"172.21.15.150:6827/2923308500","heartbeat_back_addr":"172.21.15.150:6831/2923308500","heartbeat_front_addr":"172.21.15.150:6829/2923308500","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:56:50.869427+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:57:14.694262+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:57:38.811049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:58:02.858240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:58:23.604106+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:58:45.635221+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:59:07.696960+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:59:29.976249+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.139:6801/1349154726":"2024-02-11T05:55:31.049107+0000","172.21.15.139:0/2316715134":"2024-02-11T05:55:31.049107+0000","172.21.15.139:0/4005972016":"2024-02-11T05:55:31.049107+0000","172.21.15.139:6800/1349154726":"2024-02-11T05:55:31.049107+0000","172.21.15.139:0/4231034715":"2024-02-11T05:55:31.049107+0000","172.21.15.139:6800/1743661343":"2024-02-11T05:55:08.591013+0000","172.21.15.139:0/3972026441":"2024-02-11T05:55:08.591013+0000","172.21.15.139:0/1332696927":"2024-02-11T05:55:08.591013+0000","172.21.15.139:6801/1743661343":"2024-02-11T05:55:08.591013+0000","172.21.15.139:0/687083216":"2024-02-11T05:55:08.591013+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:59:38.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:37 smithi139 ceph-mon[24727]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:38.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:37 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3138546023' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:59:38.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:37 smithi139 ceph-mon[29453]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:38.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:37 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3138546023' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:59:38.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:37 smithi150 ceph-mon[25582]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:38.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:37 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3138546023' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:59:38.464 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T05:57:41.413330+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-10T05:59:38.464 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-10T05:59:38.675 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:59:40.018 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:39 smithi139 ceph-mon[24727]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-10T05:59:40.019 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:39 smithi139 ceph-mon[29453]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-10T05:59:40.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:39 smithi150 ceph-mon[25582]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-10T05:59:40.412 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 1 2024-02-10T05:59:41.012 INFO:tasks.cephadm:Adding prometheus.a on smithi150 2024-02-10T05:59:41.012 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply prometheus '1;smithi150=a' 2024-02-10T05:59:41.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:40 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3309083861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:59:41.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:40 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3309083861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:59:41.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:40 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3309083861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:59:41.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:41 smithi150 ceph-mon[25582]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-10T05:59:42.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:41 smithi139 ceph-mon[24727]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-10T05:59:42.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:41 smithi139 ceph-mon[29453]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-10T05:59:42.902 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled prometheus update... 2024-02-10T05:59:43.496 DEBUG:teuthology.orchestra.run.smithi150:prometheus.a> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service 2024-02-10T05:59:43.498 INFO:tasks.cephadm:Adding node-exporter.a on smithi139 2024-02-10T05:59:43.498 INFO:tasks.cephadm:Adding node-exporter.b on smithi150 2024-02-10T05:59:43.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply node-exporter '2;smithi139=a;smithi150=b' 2024-02-10T05:59:44.075 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ignoring --setuser ceph since I am not root 2024-02-10T05:59:44.075 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ignoring --setgroup ceph since I am not root 2024-02-10T05:59:44.076 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:43 smithi150 ceph-mon[25582]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi150=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:44.076 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:43 smithi150 ceph-mon[25582]: Saving service prometheus spec with placement smithi150=a;count:1 2024-02-10T05:59:44.076 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:44.076 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:44.076 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:44.076 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:44.076 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:44.076 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:43 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:59:44.076 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:43 smithi150 ceph-mon[25582]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-10T05:59:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[24727]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi150=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[24727]: Saving service prometheus spec with placement smithi150=a;count:1 2024-02-10T05:59:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:59:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[24727]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-10T05:59:44.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ignoring --setuser ceph since I am not root 2024-02-10T05:59:44.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ignoring --setgroup ceph since I am not root 2024-02-10T05:59:44.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mgr[24973]: -- 172.21.15.139:0/1019511583 <== mon.1 v2:172.21.15.139:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563d1a6c2340 con 0x563d1a69a800 2024-02-10T05:59:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[29453]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi150=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[29453]: Saving service prometheus spec with placement smithi150=a;count:1 2024-02-10T05:59:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' 2024-02-10T05:59:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:59:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:43 smithi139 ceph-mon[29453]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-10T05:59:44.704 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:44 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:44.448+0000 7fc6bdae6000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:59:44.705 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:44 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:44.538+0000 7fc6bdae6000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:59:44.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:44.413+0000 7f31e2a38000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:59:44.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:44.500+0000 7f31e2a38000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:59:44.959 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:44 smithi150 ceph-mon[25582]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:59:44.960 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:44 smithi150 ceph-mon[25582]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T05:59:45.248 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:45.156+0000 7fc6bdae6000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:59:45.251 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:44 smithi139 ceph-mon[24727]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:59:45.251 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:44 smithi139 ceph-mon[24727]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T05:59:45.251 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:44 smithi139 ceph-mon[29453]: from='mgr.14150 172.21.15.139:0/1216342157' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:59:45.251 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:44 smithi139 ceph-mon[29453]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T05:59:45.251 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:45 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:45.082+0000 7f31e2a38000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:59:45.251 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:45 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:45.169+0000 7f31e2a38000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:59:45.252 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:45 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:45.250+0000 7f31e2a38000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:59:45.561 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:45.247+0000 7fc6bdae6000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:59:45.561 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:45.331+0000 7fc6bdae6000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:59:45.561 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:45.480+0000 7fc6bdae6000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:59:45.561 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:45.559+0000 7fc6bdae6000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:59:45.584 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:45 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:45.399+0000 7f31e2a38000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:59:45.584 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:45 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:45.478+0000 7f31e2a38000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:59:45.584 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:45 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:45.583+0000 7f31e2a38000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:59:45.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:45 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:45.666+0000 7f31e2a38000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:59:45.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:45 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:45.828+0000 7f31e2a38000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:59:45.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:45.666+0000 7fc6bdae6000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:59:45.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:45.748+0000 7fc6bdae6000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:59:46.258 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:45.910+0000 7fc6bdae6000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:59:46.258 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:45.993+0000 7fc6bdae6000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:59:46.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:45 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:45.912+0000 7f31e2a38000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:59:47.235 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:46.918+0000 7f31e2a38000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:59:47.235 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:47 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:47.046+0000 7f31e2a38000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:59:47.235 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:47 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:47.234+0000 7f31e2a38000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:59:47.328 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:47.010+0000 7fc6bdae6000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:59:47.328 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:47.138+0000 7fc6bdae6000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:59:47.329 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:47.327+0000 7fc6bdae6000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:59:47.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:47 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:47.320+0000 7f31e2a38000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:59:47.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:47 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:47.453+0000 7f31e2a38000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:59:47.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:47.415+0000 7fc6bdae6000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:59:47.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:47.549+0000 7fc6bdae6000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:59:48.070 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:47 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:47.768+0000 7f31e2a38000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:59:48.070 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:47 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:47.985+0000 7f31e2a38000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:59:48.070 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:48.069+0000 7f31e2a38000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:59:48.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:47.868+0000 7fc6bdae6000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:59:48.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:48.092+0000 7fc6bdae6000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:59:48.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:48.237+0000 7f31e2a38000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:59:48.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:48.176+0000 7fc6bdae6000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:59:48.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:48.346+0000 7fc6bdae6000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:59:49.433 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:49 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:49.144+0000 7f31e2a38000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:59:49.555 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:49.259+0000 7fc6bdae6000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:59:49.555 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:49.555+0000 7fc6bdae6000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:59:49.798 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:49 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:49.432+0000 7f31e2a38000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:59:49.798 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:49 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:49.527+0000 7f31e2a38000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:59:49.798 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:49 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:49.674+0000 7f31e2a38000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:59:49.818 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:49.649+0000 7fc6bdae6000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:59:49.818 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:49.801+0000 7fc6bdae6000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:59:49.818 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-mon[25582]: Active manager daemon y restarted 2024-02-10T05:59:49.818 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-mon[25582]: Activating manager daemon y 2024-02-10T05:59:49.818 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-mon[25582]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T05:59:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:49 smithi139 ceph-mon[24727]: Active manager daemon y restarted 2024-02-10T05:59:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:49 smithi139 ceph-mon[24727]: Activating manager daemon y 2024-02-10T05:59:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:49 smithi139 ceph-mon[24727]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T05:59:50.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:49 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:59:49] ENGINE Bus STARTING 2024-02-10T05:59:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:49 smithi139 ceph-mon[29453]: Active manager daemon y restarted 2024-02-10T05:59:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:49 smithi139 ceph-mon[29453]: Activating manager daemon y 2024-02-10T05:59:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:49 smithi139 ceph-mon[29453]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T05:59:50.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: [10/Feb/2024:05:59:49] ENGINE Bus STARTING 2024-02-10T05:59:50.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: CherryPy Checker: 2024-02-10T05:59:50.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: The Application mounted at '' has an empty config. 2024-02-10T05:59:50.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: 2024-02-10T05:59:50.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: [10/Feb/2024:05:59:49] ENGINE Serving on http://:::9283 2024-02-10T05:59:50.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 05:59:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: [10/Feb/2024:05:59:49] ENGINE Bus STARTED 2024-02-10T05:59:50.535 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: CherryPy Checker: 2024-02-10T05:59:50.535 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: The Application mounted at '' has an empty config. 2024-02-10T05:59:50.535 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T05:59:50.535 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:59:50] ENGINE Serving on http://:::9283 2024-02-10T05:59:50.535 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:59:50] ENGINE Bus STARTED 2024-02-10T05:59:50.535 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:59:50] ENGINE Bus STARTING 2024-02-10T05:59:50.818 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:59:50] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T05:59:50.818 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:05:59:50] ENGINE Bus STARTED 2024-02-10T05:59:50.832 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled node-exporter update... 2024-02-10T05:59:51.072 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: mgrmap e17: y(active, starting, since 0.118112s), standbys: x 2024-02-10T05:59:51.072 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:59:51.072 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:59:51.072 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:59:51.072 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: Standby manager daemon x restarted 2024-02-10T05:59:51.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: Standby manager daemon x started 2024-02-10T05:59:51.074 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:59:51.074 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:59:51.074 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:59:51.074 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:59:51.074 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: Manager daemon y is now available 2024-02-10T05:59:51.074 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:51.074 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:51.074 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:51.074 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: mgrmap e17: y(active, starting, since 0.118112s), standbys: x 2024-02-10T05:59:51.074 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:59:51.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:59:51.076 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: Standby manager daemon x restarted 2024-02-10T05:59:51.076 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: Standby manager daemon x started 2024-02-10T05:59:51.076 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:59:51.076 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:59:51.076 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:59:51.076 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:59:51.076 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: Manager daemon y is now available 2024-02-10T05:59:51.076 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:51.076 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:51.076 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:59:51.077 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:59:51.078 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:59:51.078 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:51.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: mgrmap e17: y(active, starting, since 0.118112s), standbys: x 2024-02-10T05:59:51.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:59:51.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: Standby manager daemon x restarted 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: Standby manager daemon x started 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/47250059' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: Manager daemon y is now available 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:51.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:59:51.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:59:51.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:59:51.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:59:51.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:51.846 DEBUG:teuthology.orchestra.run.smithi139:node-exporter.a> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service 2024-02-10T05:59:51.849 DEBUG:teuthology.orchestra.run.smithi150:node-exporter.b> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service 2024-02-10T05:59:51.851 INFO:tasks.cephadm:Adding alertmanager.a on smithi139 2024-02-10T05:59:51.851 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply alertmanager '1;smithi139=a' 2024-02-10T05:59:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[29453]: [10/Feb/2024:05:59:50] ENGINE Bus STARTING 2024-02-10T05:59:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[29453]: [10/Feb/2024:05:59:50] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T05:59:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[29453]: [10/Feb/2024:05:59:50] ENGINE Bus STARTED 2024-02-10T05:59:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[29453]: mgrmap e18: y(active, since 1.12522s), standbys: x 2024-02-10T05:59:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[29453]: from='client.34343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi139=a;smithi150=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[29453]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[29453]: Saving service node-exporter spec with placement smithi139=a;smithi150=b;count:2 2024-02-10T05:59:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:52.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[24727]: [10/Feb/2024:05:59:50] ENGINE Bus STARTING 2024-02-10T05:59:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[24727]: [10/Feb/2024:05:59:50] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T05:59:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[24727]: [10/Feb/2024:05:59:50] ENGINE Bus STARTED 2024-02-10T05:59:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[24727]: mgrmap e18: y(active, since 1.12522s), standbys: x 2024-02-10T05:59:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[24727]: from='client.34343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi139=a;smithi150=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[24727]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[24727]: Saving service node-exporter spec with placement smithi139=a;smithi150=b;count:2 2024-02-10T05:59:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:52.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:51 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:52.093 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:51 smithi150 ceph-mon[25582]: [10/Feb/2024:05:59:50] ENGINE Bus STARTING 2024-02-10T05:59:52.093 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:51 smithi150 ceph-mon[25582]: [10/Feb/2024:05:59:50] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T05:59:52.093 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:51 smithi150 ceph-mon[25582]: [10/Feb/2024:05:59:50] ENGINE Bus STARTED 2024-02-10T05:59:52.093 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:51 smithi150 ceph-mon[25582]: mgrmap e18: y(active, since 1.12522s), standbys: x 2024-02-10T05:59:52.093 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:51 smithi150 ceph-mon[25582]: from='client.34343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi139=a;smithi150=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:52.094 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:51 smithi150 ceph-mon[25582]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:52.094 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:51 smithi150 ceph-mon[25582]: Saving service node-exporter spec with placement smithi139=a;smithi150=b;count:2 2024-02-10T05:59:52.094 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:51 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:52.094 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:51 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:52.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:52 smithi150 ceph-mon[25582]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:52.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:52 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:52.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:52 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:52.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:52 smithi150 ceph-mon[25582]: mgrmap e19: y(active, since 2s), standbys: x 2024-02-10T05:59:52.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:52 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:52.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:52 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:53.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[29453]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:53.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:53.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:53.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[29453]: mgrmap e19: y(active, since 2s), standbys: x 2024-02-10T05:59:53.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:53.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:53.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[24727]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:53.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:53.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:53.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[24727]: mgrmap e19: y(active, since 2s), standbys: x 2024-02-10T05:59:53.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:53.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:52 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:53.931 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled alertmanager update... 2024-02-10T05:59:54.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:53 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T05:59:54.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:53 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:59:54.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:53 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T05:59:54.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:53 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:54.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:53 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:53 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T05:59:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:53 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:59:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:53 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T05:59:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:53 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:53 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:53 smithi150 ceph-mon[25582]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T05:59:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:53 smithi150 ceph-mon[25582]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:59:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:53 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T05:59:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:53 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:53 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:54.469 DEBUG:teuthology.orchestra.run.smithi139:alertmanager.a> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service 2024-02-10T05:59:54.472 INFO:tasks.cephadm:Adding grafana.a on smithi150 2024-02-10T05:59:54.472 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply grafana '1;smithi150=a' 2024-02-10T05:59:55.083 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:54 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:59:55.084 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:54 smithi150 ceph-mon[25582]: Deploying daemon prometheus.a on smithi150 2024-02-10T05:59:55.084 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:54 smithi150 ceph-mon[25582]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:55.084 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:54 smithi150 ceph-mon[25582]: from='client.34373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi139=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:55.084 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:54 smithi150 ceph-mon[25582]: Saving service alertmanager spec with placement smithi139=a;count:1 2024-02-10T05:59:55.084 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:54 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:59:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[24727]: Deploying daemon prometheus.a on smithi150 2024-02-10T05:59:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[24727]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[24727]: from='client.34373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi139=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[24727]: Saving service alertmanager spec with placement smithi139=a;count:1 2024-02-10T05:59:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:55.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:59:55.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[29453]: Deploying daemon prometheus.a on smithi150 2024-02-10T05:59:55.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[29453]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:55.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[29453]: from='client.34373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi139=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:55.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[29453]: Saving service alertmanager spec with placement smithi139=a;count:1 2024-02-10T05:59:55.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:54 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:57.096 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled grafana update... 2024-02-10T05:59:57.097 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:56 smithi150 ceph-mon[25582]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:56 smithi139 ceph-mon[24727]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:57.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:56 smithi139 ceph-mon[29453]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:58.055 DEBUG:teuthology.orchestra.run.smithi150:grafana.a> sudo journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@grafana.a.service 2024-02-10T05:59:58.058 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T05:59:58.061 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T05:59:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:58 smithi150 ceph-mon[25582]: from='client.34379 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi150=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:58 smithi150 ceph-mon[25582]: Saving service grafana spec with placement smithi150=a;count:1 2024-02-10T05:59:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:58 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 05:59:58 smithi150 ceph-mon[25582]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:58.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:58 smithi139 ceph-mon[24727]: from='client.34379 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi150=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:58.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:58 smithi139 ceph-mon[24727]: Saving service grafana spec with placement smithi150=a;count:1 2024-02-10T05:59:58.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:58 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:58.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 05:59:58 smithi139 ceph-mon[24727]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:59:58.377 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:58 smithi139 ceph-mon[29453]: from='client.34379 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi150=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:59:58.377 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:58 smithi139 ceph-mon[29453]: Saving service grafana spec with placement smithi150=a;count:1 2024-02-10T05:59:58.377 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:58 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T05:59:58.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 05:59:58 smithi139 ceph-mon[29453]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:00.178 INFO:teuthology.orchestra.run.smithi139.stdout:[client.0] 2024-02-10T06:00:00.179 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQDgEMdlrOYwChAAn14/Y0H06ElLw0j+8Eq6CA== 2024-02-10T06:00:00.988 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-02-10T06:00:00.989 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T06:00:00.989 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T06:00:01.033 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T06:00:01.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:00 smithi150 ceph-mon[25582]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:01.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:00 smithi150 ceph-mon[25582]: overall HEALTH_OK 2024-02-10T06:00:01.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:00 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3484407630' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:01.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:00 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:01.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:00 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:00:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:00 smithi139 ceph-mon[24727]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:00 smithi139 ceph-mon[24727]: overall HEALTH_OK 2024-02-10T06:00:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:00 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3484407630' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:00 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:00 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:00:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:00 smithi139 ceph-mon[29453]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:00 smithi139 ceph-mon[29453]: overall HEALTH_OK 2024-02-10T06:00:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:00 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3484407630' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:00 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:00 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:00:03.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:02 smithi150 ceph-mon[25582]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:02 smithi139 ceph-mon[24727]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:03.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:02 smithi139 ceph-mon[29453]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:04.964 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:04 smithi150 ceph-mon[25582]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:05.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:04 smithi139 ceph-mon[24727]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:05.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:04 smithi139 ceph-mon[29453]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:05.701 INFO:teuthology.orchestra.run.smithi150.stdout:[client.1] 2024-02-10T06:00:05.701 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDlEMdl9JR9KRAA0u/0PcHO5KzzbhQYmI8f5g== 2024-02-10T06:00:06.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:05 smithi139 ceph-mon[24727]: from='client.? 172.21.15.150:0/4125917005' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:06.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:05 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:06.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:05 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:00:06.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:05 smithi139 ceph-mon[29453]: from='client.? 172.21.15.150:0/4125917005' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:06.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:05 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:06.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:05 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:00:06.093 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:05 smithi150 ceph-mon[25582]: from='client.? 172.21.15.150:0/4125917005' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:06.093 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:05 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:00:06.093 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:05 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:00:06.374 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:06 smithi150 systemd[1]: Starting Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:00:06.640 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-10T06:00:06.640 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-10T06:00:06.640 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-10T06:00:06.686 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T06:00:06.686 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T06:00:06.687 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T06:00:06.911 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:06.951 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:06 smithi150 ceph-mon[25582]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:06.951 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:06 smithi150 podman[40687]: 2024-02-10 06:00:06.743632935 +0000 UTC m=+0.019166088 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-10T06:00:06.951 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:06 smithi150 podman[40687]: 2024-02-10 06:00:06.851840727 +0000 UTC m=+0.127373915 container create 00092498742fd21aec3e02c8330aec295c8c74032b293a8f5a881e0665385ba9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:00:07.032 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:06 smithi139 ceph-mon[24727]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:07.033 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:06 smithi139 ceph-mon[29453]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 podman[40687]: 2024-02-10 06:00:07.05386507 +0000 UTC m=+0.329398233 container init 00092498742fd21aec3e02c8330aec295c8c74032b293a8f5a881e0665385ba9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:00:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 podman[40687]: 2024-02-10 06:00:07.056329691 +0000 UTC m=+0.331862836 container start 00092498742fd21aec3e02c8330aec295c8c74032b293a8f5a881e0665385ba9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:00:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.080Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-10T06:00:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.080Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-10T06:00:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.080Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-10T06:00:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.080Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi150 (none))" 2024-02-10T06:00:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.080Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T06:00:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.080Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T06:00:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.081Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T06:00:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.081Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-10T06:00:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.082Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.085Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.085Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.629µs 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.085Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.085Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.085Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.363µs wal_replay_duration=118.004µs total_replay_duration=151.536µs 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.086Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.086Z caller=main.go:947 level=info msg="TSDB started" 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.086Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.101Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.396505ms db_storage=649ns remote_storage=1.207µs web_handler=303ns query_engine=418ns scrape=374.941µs scrape_sd=16.737µs notify=1.114µs notify_sd=1.467µs rules=14.782608ms 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:00:07.101Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 bash[40687]: 00092498742fd21aec3e02c8330aec295c8c74032b293a8f5a881e0665385ba9 2024-02-10T06:00:07.376 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:00:07 smithi150 systemd[1]: Started Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:00:08.322 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:08 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:08.322 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:08 smithi139 ceph-mon[24727]: Deploying daemon node-exporter.a on smithi139 2024-02-10T06:00:08.322 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:08 smithi139 ceph-mon[24727]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:08.322 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:08 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:08.323 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:08 smithi139 ceph-mon[29453]: Deploying daemon node-exporter.a on smithi139 2024-02-10T06:00:08.323 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:08 smithi139 ceph-mon[29453]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:08 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:08 smithi150 ceph-mon[25582]: Deploying daemon node-exporter.a on smithi139 2024-02-10T06:00:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:08 smithi150 ceph-mon[25582]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:09.000 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:00:09.859 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":19,"active_gid":24334,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1583722991},{"type":"v1","addr":"172.21.15.139:6801","nonce":1583722991}]},"active_addr":"172.21.15.139:6801/1583722991","active_change":"2024-02-10T05:59:49.689532+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34334,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.139:8443/","prometheus":"http://172.21.15.139:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":1839384734}]},{"addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":150105896}]},{"addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":3505229686}]},{"addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":3021323360}]}]}} 2024-02-10T06:00:09.863 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T06:00:09.863 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T06:00:09.863 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T06:00:10.082 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:09 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/4241053769' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:00:10.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:09 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/4241053769' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:00:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:09 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/4241053769' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:00:11.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:10 smithi150 ceph-mon[25582]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:11.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:10 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:11.142 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:10 smithi139 ceph-mon[24727]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:11.143 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:10 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:11.143 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:10 smithi139 ceph-mon[29453]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:11.143 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:10 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:11.586 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:11 smithi139 bash[44311]: Writing manifest to image destination 2024-02-10T06:00:12.002 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:11 smithi139 podman[44311]: 2024-02-10 06:00:11.675077225 +0000 UTC m=+2.878843305 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-10T06:00:12.283 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 podman[44311]: 2024-02-10 06:00:12.284202081 +0000 UTC m=+3.487968143 container create 5b38421d9397c150faab6ef5b0cb4d663013713a6b6c63b4727f1e28f5178e85 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:00:12.552 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 podman[44311]: 2024-02-10 06:00:12.461167142 +0000 UTC m=+3.664933208 container init 5b38421d9397c150faab6ef5b0cb4d663013713a6b6c63b4727f1e28f5178e85 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:00:12.553 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 podman[44311]: 2024-02-10 06:00:12.463793496 +0000 UTC m=+3.667559560 container start 5b38421d9397c150faab6ef5b0cb4d663013713a6b6c63b4727f1e28f5178e85 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:00:12.553 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.467Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T06:00:12.553 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.467Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T06:00:12.553 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T06:00:12.553 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T06:00:12.553 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T06:00:12.553 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T06:00:12.553 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T06:00:12.553 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T06:00:12.553 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T06:00:12.553 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T06:00:12.554 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T06:00:12.555 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T06:00:12.556 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T06:00:12.556 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T06:00:12.556 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T06:00:12.556 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T06:00:12.556 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T06:00:12.556 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T06:00:12.556 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T06:00:12.556 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T06:00:12.556 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T06:00:12.556 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T06:00:12.557 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.468Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T06:00:12.557 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[44488]: ts=2024-02-10T06:00:12.469Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T06:00:12.557 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 bash[44311]: 5b38421d9397c150faab6ef5b0cb4d663013713a6b6c63b4727f1e28f5178e85 2024-02-10T06:00:12.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:00:12 smithi139 systemd[1]: Started Ceph node-exporter.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:00:12.927 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:12 smithi150 ceph-mon[25582]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:12.927 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:12 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:13.292 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:00:13.292 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":59,"fsid":"ce44e742-c7d8-11ee-95b8-87774f69a715","created":"2024-02-10T05:54:37.242368+0000","modified":"2024-02-10T05:59:49.689209+0000","last_up_change":"2024-02-10T05:59:31.819060+0000","last_in_change":"2024-02-10T05:59:14.928470+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:57:41.413330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6803","nonce":3875590395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6805","nonce":3875590395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6809","nonce":3875590395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6807","nonce":3875590395}]},"public_addr":"172.21.15.139:6803/3875590395","cluster_addr":"172.21.15.139:6805/3875590395","heartbeat_back_addr":"172.21.15.139:6809/3875590395","heartbeat_front_addr":"172.21.15.139:6807/3875590395","state":["exists","up"]},{"osd":1,"uuid":"a39bbeb2-0fd3-4584-bde1-2578c452924e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6811","nonce":2796500287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6813","nonce":2796500287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6817","nonce":2796500287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6815","nonce":2796500287}]},"public_addr":"172.21.15.139:6811/2796500287","cluster_addr":"172.21.15.139:6813/2796500287","heartbeat_back_addr":"172.21.15.139:6817/2796500287","heartbeat_front_addr":"172.21.15.139:6815/2796500287","state":["exists","up"]},{"osd":2,"uuid":"1f801d19-78c0-40e6-8f36-efd49380be35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6819","nonce":872620670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6821","nonce":872620670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6825","nonce":872620670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6823","nonce":872620670}]},"public_addr":"172.21.15.139:6819/872620670","cluster_addr":"172.21.15.139:6821/872620670","heartbeat_back_addr":"172.21.15.139:6825/872620670","heartbeat_front_addr":"172.21.15.139:6823/872620670","state":["exists","up"]},{"osd":3,"uuid":"433e8787-cfe9-4309-b185-473bc2d38f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6827","nonce":684450120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6829","nonce":684450120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6833","nonce":684450120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6831","nonce":684450120}]},"public_addr":"172.21.15.139:6827/684450120","cluster_addr":"172.21.15.139:6829/684450120","heartbeat_back_addr":"172.21.15.139:6833/684450120","heartbeat_front_addr":"172.21.15.139:6831/684450120","state":["exists","up"]},{"osd":4,"uuid":"66b23be8-0d14-4bf6-98cc-ee09db996d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6801","nonce":290935797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6803","nonce":290935797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6807","nonce":290935797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6805","nonce":290935797}]},"public_addr":"172.21.15.150:6801/290935797","cluster_addr":"172.21.15.150:6803/290935797","heartbeat_back_addr":"172.21.15.150:6807/290935797","heartbeat_front_addr":"172.21.15.150:6805/290935797","state":["exists","up"]},{"osd":5,"uuid":"0ffbdbb7-af3d-4023-960e-a61b22010132","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6809","nonce":1581167176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6811","nonce":1581167176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6815","nonce":1581167176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6813","nonce":1581167176}]},"public_addr":"172.21.15.150:6809/1581167176","cluster_addr":"172.21.15.150:6811/1581167176","heartbeat_back_addr":"172.21.15.150:6815/1581167176","heartbeat_front_addr":"172.21.15.150:6813/1581167176","state":["exists","up"]},{"osd":6,"uuid":"aa1bab2a-38b3-429a-ba5a-ff7499fe9b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6817","nonce":382604718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6819","nonce":382604718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6823","nonce":382604718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6821","nonce":382604718}]},"public_addr":"172.21.15.150:6817/382604718","cluster_addr":"172.21.15.150:6819/382604718","heartbeat_back_addr":"172.21.15.150:6823/382604718","heartbeat_front_addr":"172.21.15.150:6821/382604718","state":["exists","up"]},{"osd":7,"uuid":"309c3600-54dd-4102-b5a6-2080aba9b681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6825","nonce":2923308500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6827","nonce":2923308500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6831","nonce":2923308500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6829","nonce":2923308500}]},"public_addr":"172.21.15.150:6825/2923308500","cluster_addr":"172.21.15.150:6827/2923308500","heartbeat_back_addr":"172.21.15.150:6831/2923308500","heartbeat_front_addr":"172.21.15.150:6829/2923308500","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:56:50.869427+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:57:14.694262+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:57:38.811049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:58:02.858240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:58:23.604106+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:58:45.635221+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:59:07.696960+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:59:29.976249+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.139:0/2159038905":"2024-02-11T05:59:49.689174+0000","172.21.15.139:6800/3587488312":"2024-02-11T05:59:49.689174+0000","172.21.15.139:0/968128567":"2024-02-11T05:59:49.689174+0000","172.21.15.139:6801/3587488312":"2024-02-11T05:59:49.689174+0000","172.21.15.139:0/347598603":"2024-02-11T05:59:49.689174+0000","172.21.15.139:6801/1349154726":"2024-02-11T05:55:31.049107+0000","172.21.15.139:0/2316715134":"2024-02-11T05:55:31.049107+0000","172.21.15.139:0/4005972016":"2024-02-11T05:55:31.049107+0000","172.21.15.139:6800/1349154726":"2024-02-11T05:55:31.049107+0000","172.21.15.139:0/4231034715":"2024-02-11T05:55:31.049107+0000","172.21.15.139:6800/1743661343":"2024-02-11T05:55:08.591013+0000","172.21.15.139:0/3393873192":"2024-02-11T05:59:49.689174+0000","172.21.15.139:0/3972026441":"2024-02-11T05:55:08.591013+0000","172.21.15.139:0/1332696927":"2024-02-11T05:55:08.591013+0000","172.21.15.139:6801/1743661343":"2024-02-11T05:55:08.591013+0000","172.21.15.139:0/687083216":"2024-02-11T05:55:08.591013+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:00:13.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-mon[24727]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:13.293 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:13.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-mon[29453]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:13.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:12 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:13.624 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:13 smithi150 bash[40930]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-10T06:00:14.043 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:13 smithi139 ceph-mon[24727]: Deploying daemon node-exporter.b on smithi150 2024-02-10T06:00:14.043 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:13 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3174910741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:00:14.043 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:13 smithi139 ceph-mon[29453]: Deploying daemon node-exporter.b on smithi150 2024-02-10T06:00:14.043 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:13 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3174910741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:00:14.068 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T06:00:14.068 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T06:00:14.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:13 smithi150 ceph-mon[25582]: Deploying daemon node-exporter.b on smithi150 2024-02-10T06:00:14.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:13 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3174910741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:00:14.124 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:13 smithi150 bash[40930]: Getting image source signatures 2024-02-10T06:00:14.124 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:13 smithi150 bash[40930]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-10T06:00:14.124 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:13 smithi150 bash[40930]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-10T06:00:14.125 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:13 smithi150 bash[40930]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-10T06:00:14.285 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:15.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:14 smithi139 ceph-mon[24727]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:15.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:14 smithi139 ceph-mon[29453]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:15.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:14 smithi150 ceph-mon[25582]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:15.624 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:15 smithi150 bash[40930]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-10T06:00:15.624 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:15 smithi150 bash[40930]: Writing manifest to image destination 2024-02-10T06:00:16.073 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:15 smithi150 podman[40930]: 2024-02-10 06:00:15.795798316 +0000 UTC m=+2.403977147 container create c79c2778bd87546a8060707c1b8a89b4d8711c3404a139264c4f2feff593567e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:00:16.073 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:15 smithi150 podman[40930]: 2024-02-10 06:00:15.722208593 +0000 UTC m=+2.330387430 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-10T06:00:16.073 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:15 smithi150 podman[40930]: 2024-02-10 06:00:15.998071239 +0000 UTC m=+2.606250075 container init c79c2778bd87546a8060707c1b8a89b4d8711c3404a139264c4f2feff593567e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:00:16.073 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 podman[40930]: 2024-02-10 06:00:16.00085587 +0000 UTC m=+2.609034703 container start c79c2778bd87546a8060707c1b8a89b4d8711c3404a139264c4f2feff593567e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.004Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.004Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.004Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.004Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T06:00:16.074 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T06:00:16.075 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.005Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T06:00:16.076 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T06:00:16.076 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T06:00:16.076 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T06:00:16.076 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T06:00:16.076 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T06:00:16.076 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T06:00:16.076 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T06:00:16.076 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T06:00:16.076 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T06:00:16.077 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[40982]: ts=2024-02-10T06:00:16.006Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T06:00:16.078 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 bash[40930]: c79c2778bd87546a8060707c1b8a89b4d8711c3404a139264c4f2feff593567e 2024-02-10T06:00:16.152 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:00:16.152 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":59,"fsid":"ce44e742-c7d8-11ee-95b8-87774f69a715","created":"2024-02-10T05:54:37.242368+0000","modified":"2024-02-10T05:59:49.689209+0000","last_up_change":"2024-02-10T05:59:31.819060+0000","last_in_change":"2024-02-10T05:59:14.928470+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:57:41.413330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ac8a7632-19f3-4e47-b9e8-c114bf5d3f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6803","nonce":3875590395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6805","nonce":3875590395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6809","nonce":3875590395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":3875590395},{"type":"v1","addr":"172.21.15.139:6807","nonce":3875590395}]},"public_addr":"172.21.15.139:6803/3875590395","cluster_addr":"172.21.15.139:6805/3875590395","heartbeat_back_addr":"172.21.15.139:6809/3875590395","heartbeat_front_addr":"172.21.15.139:6807/3875590395","state":["exists","up"]},{"osd":1,"uuid":"a39bbeb2-0fd3-4584-bde1-2578c452924e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6811","nonce":2796500287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6813","nonce":2796500287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6817","nonce":2796500287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2796500287},{"type":"v1","addr":"172.21.15.139:6815","nonce":2796500287}]},"public_addr":"172.21.15.139:6811/2796500287","cluster_addr":"172.21.15.139:6813/2796500287","heartbeat_back_addr":"172.21.15.139:6817/2796500287","heartbeat_front_addr":"172.21.15.139:6815/2796500287","state":["exists","up"]},{"osd":2,"uuid":"1f801d19-78c0-40e6-8f36-efd49380be35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6819","nonce":872620670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6821","nonce":872620670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6825","nonce":872620670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":872620670},{"type":"v1","addr":"172.21.15.139:6823","nonce":872620670}]},"public_addr":"172.21.15.139:6819/872620670","cluster_addr":"172.21.15.139:6821/872620670","heartbeat_back_addr":"172.21.15.139:6825/872620670","heartbeat_front_addr":"172.21.15.139:6823/872620670","state":["exists","up"]},{"osd":3,"uuid":"433e8787-cfe9-4309-b185-473bc2d38f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6827","nonce":684450120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6829","nonce":684450120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6833","nonce":684450120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":684450120},{"type":"v1","addr":"172.21.15.139:6831","nonce":684450120}]},"public_addr":"172.21.15.139:6827/684450120","cluster_addr":"172.21.15.139:6829/684450120","heartbeat_back_addr":"172.21.15.139:6833/684450120","heartbeat_front_addr":"172.21.15.139:6831/684450120","state":["exists","up"]},{"osd":4,"uuid":"66b23be8-0d14-4bf6-98cc-ee09db996d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6801","nonce":290935797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6803","nonce":290935797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6807","nonce":290935797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":290935797},{"type":"v1","addr":"172.21.15.150:6805","nonce":290935797}]},"public_addr":"172.21.15.150:6801/290935797","cluster_addr":"172.21.15.150:6803/290935797","heartbeat_back_addr":"172.21.15.150:6807/290935797","heartbeat_front_addr":"172.21.15.150:6805/290935797","state":["exists","up"]},{"osd":5,"uuid":"0ffbdbb7-af3d-4023-960e-a61b22010132","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6809","nonce":1581167176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6811","nonce":1581167176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6815","nonce":1581167176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1581167176},{"type":"v1","addr":"172.21.15.150:6813","nonce":1581167176}]},"public_addr":"172.21.15.150:6809/1581167176","cluster_addr":"172.21.15.150:6811/1581167176","heartbeat_back_addr":"172.21.15.150:6815/1581167176","heartbeat_front_addr":"172.21.15.150:6813/1581167176","state":["exists","up"]},{"osd":6,"uuid":"aa1bab2a-38b3-429a-ba5a-ff7499fe9b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6817","nonce":382604718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6819","nonce":382604718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6823","nonce":382604718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":382604718},{"type":"v1","addr":"172.21.15.150:6821","nonce":382604718}]},"public_addr":"172.21.15.150:6817/382604718","cluster_addr":"172.21.15.150:6819/382604718","heartbeat_back_addr":"172.21.15.150:6823/382604718","heartbeat_front_addr":"172.21.15.150:6821/382604718","state":["exists","up"]},{"osd":7,"uuid":"309c3600-54dd-4102-b5a6-2080aba9b681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6825","nonce":2923308500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6827","nonce":2923308500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6831","nonce":2923308500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2923308500},{"type":"v1","addr":"172.21.15.150:6829","nonce":2923308500}]},"public_addr":"172.21.15.150:6825/2923308500","cluster_addr":"172.21.15.150:6827/2923308500","heartbeat_back_addr":"172.21.15.150:6831/2923308500","heartbeat_front_addr":"172.21.15.150:6829/2923308500","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:56:50.869427+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:57:14.694262+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:57:38.811049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:58:02.858240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:58:23.604106+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:58:45.635221+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:59:07.696960+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:59:29.976249+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.139:0/2159038905":"2024-02-11T05:59:49.689174+0000","172.21.15.139:6800/3587488312":"2024-02-11T05:59:49.689174+0000","172.21.15.139:0/968128567":"2024-02-11T05:59:49.689174+0000","172.21.15.139:6801/3587488312":"2024-02-11T05:59:49.689174+0000","172.21.15.139:0/347598603":"2024-02-11T05:59:49.689174+0000","172.21.15.139:6801/1349154726":"2024-02-11T05:55:31.049107+0000","172.21.15.139:0/2316715134":"2024-02-11T05:55:31.049107+0000","172.21.15.139:0/4005972016":"2024-02-11T05:55:31.049107+0000","172.21.15.139:6800/1349154726":"2024-02-11T05:55:31.049107+0000","172.21.15.139:0/4231034715":"2024-02-11T05:55:31.049107+0000","172.21.15.139:6800/1743661343":"2024-02-11T05:55:08.591013+0000","172.21.15.139:0/3393873192":"2024-02-11T05:59:49.689174+0000","172.21.15.139:0/3972026441":"2024-02-11T05:55:08.591013+0000","172.21.15.139:0/1332696927":"2024-02-11T05:55:08.591013+0000","172.21.15.139:6801/1743661343":"2024-02-11T05:55:08.591013+0000","172.21.15.139:0/687083216":"2024-02-11T05:55:08.591013+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:00:16.374 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:00:16 smithi150 systemd[1]: Started Ceph node-exporter.b for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:00:16.893 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T06:00:16.894 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T06:00:16.894 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T06:00:16.895 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T06:00:16.895 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T06:00:16.896 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T06:00:16.896 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-10T06:00:16.896 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-10T06:00:17.101 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-mon[25582]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:17.102 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:17.102 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:00:17.102 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:00:17.102 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:00:17.102 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:16 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/178227146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:00:17.213 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[24727]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:17.213 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:17.213 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:00:17.213 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:00:17.213 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:00:17.213 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/178227146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:00:17.213 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[29453]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:17.213 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:17.213 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:00:17.213 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:00:17.214 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:00:17.214 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:16 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/178227146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:00:17.674 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:17.681 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:17.684 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:17.684 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:17.687 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:17.688 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:17.702 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:17.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:00:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:00:19.320 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:18 smithi139 ceph-mon[24727]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:19.320 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:18 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:19.320 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:18 smithi139 ceph-mon[29453]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:19.321 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:18 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:18 smithi150 ceph-mon[25582]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:18 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:20.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:00:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:00:20] "GET /metrics HTTP/1.1" 200 191522 "" "Prometheus/2.33.4" 2024-02-10T06:00:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:20 smithi150 ceph-mon[25582]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:20 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:21.137 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:20 smithi139 ceph-mon[24727]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:21.137 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:20 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:21.138 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:20 smithi139 ceph-mon[29453]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:21.138 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:20 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:21.367 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:23.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:22 smithi139 ceph-mon[24727]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:23.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:22 smithi139 ceph-mon[29453]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:23.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:22 smithi150 ceph-mon[25582]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:24.994 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:24 smithi139 ceph-mon[24727]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:24.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:24 smithi139 ceph-mon[29453]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:25.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:24 smithi150 ceph-mon[25582]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:26.892 INFO:teuthology.orchestra.run.smithi139.stdout:55834574888 2024-02-10T06:00:26.893 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T06:00:26.919 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:26 smithi139 ceph-mon[24727]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:26.919 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:26 smithi139 ceph-mon[29453]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:27.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:26 smithi150 ceph-mon[25582]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:28.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:00:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:00:28.180 INFO:teuthology.orchestra.run.smithi139.stdout:163208757270 2024-02-10T06:00:28.180 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T06:00:29.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:28 smithi139 ceph-mon[29453]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:29.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:28 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:29.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:28 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:29.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:28 smithi139 ceph-mon[24727]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:29.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:28 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:29.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:28 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:29.309 INFO:teuthology.orchestra.run.smithi139.stdout:107374182430 2024-02-10T06:00:29.310 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T06:00:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:28 smithi150 ceph-mon[25582]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:28 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:28 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:30.251 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:29 smithi139 ceph-mon[24727]: Deploying daemon alertmanager.a on smithi139 2024-02-10T06:00:30.252 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:29 smithi139 ceph-mon[29453]: Deploying daemon alertmanager.a on smithi139 2024-02-10T06:00:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:29 smithi150 ceph-mon[25582]: Deploying daemon alertmanager.a on smithi139 2024-02-10T06:00:30.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:00:30 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:00:30] "GET /metrics HTTP/1.1" 200 191522 "" "Prometheus/2.33.4" 2024-02-10T06:00:31.038 INFO:teuthology.orchestra.run.smithi139.stdout:77309411364 2024-02-10T06:00:31.038 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T06:00:31.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:30 smithi139 ceph-mon[24727]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:31.100 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:30 smithi139 ceph-mon[29453]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:30 smithi150 ceph-mon[25582]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:32.732 INFO:teuthology.orchestra.run.smithi139.stdout:128849018907 2024-02-10T06:00:32.732 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T06:00:33.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:32 smithi139 ceph-mon[24727]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:33.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:32 smithi139 ceph-mon[29453]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:32 smithi150 ceph-mon[25582]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:34.154 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:34.154 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:34.156 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:34.157 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:34.891 INFO:teuthology.orchestra.run.smithi139.stdout:231928233998 2024-02-10T06:00:34.891 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-10T06:00:34.916 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:34 smithi139 ceph-mon[24727]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:34.916 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:34 smithi139 ceph-mon[29453]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:35.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:34 smithi150 ceph-mon[25582]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:35.662 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:37.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:36 smithi150 ceph-mon[25582]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:37.288 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:36 smithi139 ceph-mon[24727]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:37.289 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:36 smithi139 ceph-mon[29453]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:38.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:00:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:00:38.453 INFO:teuthology.orchestra.run.smithi139.stdout:197568495636 2024-02-10T06:00:38.454 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-10T06:00:39.161 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:38 smithi139 ceph-mon[24727]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:39.161 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:38 smithi139 ceph-mon[29453]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:39.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:38 smithi150 ceph-mon[25582]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:40.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:00:40 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:00:40] "GET /metrics HTTP/1.1" 200 191517 "" "Prometheus/2.33.4" 2024-02-10T06:00:40.930 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:40.950 INFO:teuthology.orchestra.run.smithi139.stdout:34359738414 2024-02-10T06:00:40.950 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T06:00:41.249 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:40 smithi139 ceph-mon[24727]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:41.249 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:40 smithi139 ceph-mon[29453]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:40 smithi150 ceph-mon[25582]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:42.192 INFO:teuthology.orchestra.run.smithi139.stdout:163208757273 2024-02-10T06:00:43.022 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:42 smithi139 ceph-mon[24727]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:43.022 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:42 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3371570018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T06:00:43.023 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:42 smithi139 ceph-mon[29453]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:43.023 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:42 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3371570018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T06:00:43.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:42 smithi150 ceph-mon[25582]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:43.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:42 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3371570018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T06:00:44.471 INFO:teuthology.orchestra.run.smithi139.stdout:128849018909 2024-02-10T06:00:45.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:44 smithi139 ceph-mon[24727]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:45.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:44 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1278429421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T06:00:45.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:44 smithi139 ceph-mon[29453]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:45.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:44 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1278429421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T06:00:45.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:44 smithi150 ceph-mon[25582]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:45.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:44 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1278429421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T06:00:47.064 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:46 smithi139 ceph-mon[24727]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:47.064 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:46 smithi139 ceph-mon[29453]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:47.258 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:46 smithi150 ceph-mon[25582]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:47.386 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018907 got 128849018909 for osd.4 2024-02-10T06:00:47.386 DEBUG:teuthology.parallel:result is None 2024-02-10T06:00:47.703 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:47.704 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:47.729 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757270 got 163208757273 for osd.5 2024-02-10T06:00:47.729 DEBUG:teuthology.parallel:result is None 2024-02-10T06:00:48.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:00:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:00:48.562 INFO:teuthology.orchestra.run.smithi139.stdout:77309411368 2024-02-10T06:00:49.081 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:48 smithi139 ceph-mon[24727]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:49.081 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:48 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/4248264685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T06:00:49.081 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:48 smithi139 ceph-mon[29453]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:49.081 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:48 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/4248264685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T06:00:49.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:48 smithi150 ceph-mon[25582]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:49.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:48 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/4248264685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T06:00:49.944 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411364 got 77309411368 for osd.2 2024-02-10T06:00:49.945 DEBUG:teuthology.parallel:result is None 2024-02-10T06:00:50.317 INFO:teuthology.orchestra.run.smithi139.stdout:107374182434 2024-02-10T06:00:50.883 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:00:50] "GET /metrics HTTP/1.1" 200 191517 "" "Prometheus/2.33.4" 2024-02-10T06:00:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[24727]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:00:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:00:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:00:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:00:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2433050567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T06:00:51.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[29453]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:51.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:00:51.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:00:51.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:00:51.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:00:51.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:50 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2433050567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T06:00:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:50 smithi150 ceph-mon[25582]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:00:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:00:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:00:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:00:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:50 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2433050567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T06:00:52.153 INFO:teuthology.orchestra.run.smithi139.stdout:55834574894 2024-02-10T06:00:52.804 INFO:teuthology.orchestra.run.smithi139.stdout:231928234003 2024-02-10T06:00:53.062 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:52 smithi139 ceph-mon[24727]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:53.063 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:52 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1781858686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T06:00:53.063 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:52 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1075228940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T06:00:53.064 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:52 smithi139 ceph-mon[29453]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:53.064 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:52 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1781858686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T06:00:53.064 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:52 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1075228940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T06:00:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:52 smithi150 ceph-mon[25582]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:52 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1781858686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T06:00:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:52 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1075228940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T06:00:53.784 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182430 got 107374182434 for osd.3 2024-02-10T06:00:53.785 DEBUG:teuthology.parallel:result is None 2024-02-10T06:00:54.203 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233998 got 231928234003 for osd.7 2024-02-10T06:00:54.204 DEBUG:teuthology.parallel:result is None 2024-02-10T06:00:55.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:54 smithi139 ceph-mon[24727]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:55.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:54 smithi139 ceph-mon[29453]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:55.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:54 smithi150 ceph-mon[25582]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:55.347 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574894 for osd.1 2024-02-10T06:00:55.347 DEBUG:teuthology.parallel:result is None 2024-02-10T06:00:55.705 INFO:teuthology.orchestra.run.smithi139.stdout:197568495639 2024-02-10T06:00:56.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:55 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1979642053' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T06:00:56.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:55 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1979642053' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T06:00:56.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:55 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1979642053' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T06:00:56.394 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495636 got 197568495639 for osd.6 2024-02-10T06:00:56.394 DEBUG:teuthology.parallel:result is None 2024-02-10T06:00:57.031 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:56 smithi139 ceph-mon[24727]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:57.031 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:56 smithi139 ceph-mon[29453]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:57.111 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:56 smithi150 ceph-mon[25582]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:57.308 INFO:teuthology.orchestra.run.smithi139.stdout:34359738419 2024-02-10T06:00:57.968 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:57.968 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:57.968 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:57.968 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:00:57.968 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:57.968 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/4257969571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T06:00:57.969 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:57.969 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:57.969 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:57.969 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:00:57.969 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:57.969 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:57 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/4257969571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T06:00:57.992 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738419 for osd.0 2024-02-10T06:00:57.993 DEBUG:teuthology.parallel:result is None 2024-02-10T06:00:57.993 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T06:00:57.993 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T06:00:58.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:00:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:00:58.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:57 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:58.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:57 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:58.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:57 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:58.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:57 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:00:58.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:57 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:00:58.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:57 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/4257969571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T06:00:58.216 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:00:58.946 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:00:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[47532]: level=info ts=2024-02-10T06:00:58.796Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000282141s 2024-02-10T06:00:59.299 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:58 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:00:59.300 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:58 smithi139 ceph-mon[24727]: Deploying daemon grafana.a on smithi150 2024-02-10T06:00:59.300 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:00:58 smithi139 ceph-mon[24727]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:59.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:58 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:00:59.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:58 smithi139 ceph-mon[29453]: Deploying daemon grafana.a on smithi150 2024-02-10T06:00:59.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:00:58 smithi139 ceph-mon[29453]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:58 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:00:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:58 smithi150 ceph-mon[25582]: Deploying daemon grafana.a on smithi150 2024-02-10T06:00:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:00:58 smithi150 ceph-mon[25582]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:59.938 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:00:59.939 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2024-02-10T06:01:00.672 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-02-10T06:00:59.824261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48912,"kb_used_data":6096,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912304,"statfs":{"total":767960285184,"available":767910199296,"internally_reserved":0,"allocated":6242304,"data_stored":3488982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001890"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-02-10T05:59:49.892432+0000","last_change":"2024-02-10T05:59:35.863272+0000","last_active":"2024-02-10T05:59:49.892432+0000","last_peered":"2024-02-10T05:59:49.892432+0000","last_clean":"2024-02-10T05:59:49.892432+0000","last_became_active":"2024-02-10T05:59:35.862782+0000","last_became_peered":"2024-02-10T05:59:35.862782+0000","last_unstale":"2024-02-10T05:59:49.892432+0000","last_undegraded":"2024-02-10T05:59:49.892432+0000","last_fullsized":"2024-02-10T05:59:49.892432+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:41.599812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:41.599812+0000","last_clean_scrub_stamp":"2024-02-10T05:57:41.599812+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:52:42.267399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.61799999999999999}]},{"osd":1,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.71699999999999997}]},{"osd":2,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.76100000000000001}]},{"osd":3,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.73599999999999999}]},{"osd":4,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.57499999999999996}]},{"osd":5,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.68000000000000005}]},{"osd":6,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.749},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.70599999999999996}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.60299999999999998}]},{"osd":1,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56399999999999995}]},{"osd":3,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.53100000000000003}]},{"osd":4,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.41599999999999998}]},{"osd":5,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.36399999999999999}]},{"osd":7,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.52200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6328,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738824,"statfs":{"total":95995035648,"available":95988555776,"internally_reserved":0,"allocated":573440,"data_stored":237311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.32300000000000001,"15min":0.30299999999999999},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.35399999999999998,"15min":0.33600000000000002},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56100000000000005,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.437}]},{"osd":2,"last update":"Sat Feb 10 06:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.40200000000000002,"15min":0.40600000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55400000000000005,"5min":0.626,"15min":0.626},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38800000000000001,"15min":0.39200000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.495,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.441,"15min":0.441},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.441,"15min":0.441},"min":{"1min":0.34200000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.54700000000000004,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.48599999999999999}]},{"osd":4,"last update":"Sat Feb 10 06:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76600000000000001,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56200000000000006,"15min":0.56699999999999995},"min":{"1min":0.42399999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.627}]},{"osd":5,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.55600000000000005},"min":{"1min":0.44400000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.70699999999999996,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.70599999999999996,"5min":0.751,"15min":0.751},"last":0.63700000000000001}]},{"osd":6,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.75700000000000001}]},{"osd":7,"last update":"Sat Feb 10 06:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.66000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 06:00:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.316},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.48099999999999998,"5min":0.504,"15min":0.504},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.32800000000000001},"min":{"1min":0.25800000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.48999999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.249}]},{"osd":2,"last update":"Sat Feb 10 06:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.36899999999999999,"15min":0.372},"min":{"1min":0.26300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.50700000000000001,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.41099999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.503,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.26600000000000001}]},{"osd":3,"last update":"Sat Feb 10 06:00:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.44900000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57399999999999995,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.443,"15min":0.442},"min":{"1min":0.34499999999999997,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.40799999999999997}]},{"osd":4,"last update":"Sat Feb 10 06:00:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.83299999999999996,"15min":0.876},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80300000000000005,"5min":7.3040000000000003,"15min":7.3040000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.44600000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.755,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.53100000000000003}]},{"osd":5,"last update":"Sat Feb 10 06:00:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.48699999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.76900000000000002,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.441}]},{"osd":6,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.48599999999999999}]},{"osd":7,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.60199999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36599999999999999,"15min":0.36699999999999999},"min":{"1min":0.221,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53400000000000003,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.377},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.627,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.33900000000000002}]},{"osd":1,"last update":"Sat Feb 10 06:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41099999999999998,"15min":0.40899999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42199999999999999,"15min":0.42499999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Feb 10 06:00:09 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.45100000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.438,"15min":0.44},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56000000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Feb 10 06:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.49099999999999999}]},{"osd":5,"last update":"Sat Feb 10 06:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.628},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.69399999999999995}]},{"osd":6,"last update":"Sat Feb 10 06:00:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.70199999999999996}]},{"osd":7,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.51800000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.37,"15min":0.36899999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.497},{"interface":"front","average":{"1min":0.438,"5min":0.41899999999999998,"15min":0.41599999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.59099999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.44900000000000001}]},{"osd":1,"last update":"Sat Feb 10 06:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.59599999999999997,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.47999999999999998}]},{"osd":2,"last update":"Sat Feb 10 06:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.29799999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65000000000000002,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42999999999999999,"15min":0.432},"min":{"1min":0.314,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.57099999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.42799999999999999}]},{"osd":4,"last update":"Sat Feb 10 06:00:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60299999999999998,"15min":0.60699999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58799999999999997,"15min":0.59199999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.71999999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.59899999999999998}]},{"osd":5,"last update":"Sat Feb 10 06:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.623},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.64500000000000002}]},{"osd":6,"last update":"Sat Feb 10 06:00:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.56599999999999995}]},{"osd":7,"last update":"Sat Feb 10 06:00:33 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.69199999999999995}]}]},{"osd":4,"up_from":30,"seq":128849018912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55700000000000005,"15min":0.56399999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69699999999999995,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.443},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.35599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.35699999999999998}]},{"osd":1,"last update":"Sat Feb 10 06:00:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72299999999999998,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72399999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Feb 10 06:00:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.75,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71099999999999997,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.432}]},{"osd":3,"last update":"Sat Feb 10 06:00:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.372,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Sat Feb 10 06:00:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.41799999999999998,"15min":0.40999999999999998},"min":{"1min":0.309,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.371},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.50700000000000001,"15min":0.49099999999999999},"min":{"1min":0.41799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.52400000000000002}]},{"osd":6,"last update":"Sat Feb 10 06:00:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.59299999999999997}]},{"osd":7,"last update":"Sat Feb 10 06:00:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.53900000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.71799999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.51400000000000001,"15min":0.50600000000000001},"min":{"1min":0.39600000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.73299999999999998,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.60499999999999998}]},{"osd":1,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.35899999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69699999999999995,"5min":0.749,"15min":0.749},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76000000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.72399999999999998}]},{"osd":2,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.42299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.54800000000000004},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.73999999999999999,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.70299999999999996}]},{"osd":3,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69899999999999995,"5min":0.746,"15min":0.746},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.53700000000000003},"min":{"1min":0.41899999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.75}]},{"osd":4,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.45400000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.47799999999999998,"15min":0.46600000000000003},"min":{"1min":0.36799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.56399999999999995}]},{"osd":6,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.68999999999999995}]},{"osd":7,"last update":"Sat Feb 10 06:00:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T06:01:00.675 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T06:01:00.879 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:01:00 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:01:00] "GET /metrics HTTP/1.1" 200 191517 "" "Prometheus/2.33.4" 2024-02-10T06:01:00.893 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:01:01.146 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:00 smithi139 ceph-mon[24727]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:01.146 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:00 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:01.146 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:00 smithi139 ceph-mon[24727]: from='client.24505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:01:01.146 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:00 smithi139 ceph-mon[29453]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:01.146 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:00 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:01.147 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:00 smithi139 ceph-mon[29453]: from='client.24505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:01:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:00 smithi150 ceph-mon[25582]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:00 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:00 smithi150 ceph-mon[25582]: from='client.24505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:01:02.813 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:01:02.814 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2024-02-10T06:01:03.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:02 smithi139 ceph-mon[24727]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:03.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:02 smithi139 ceph-mon[29453]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:02 smithi150 ceph-mon[25582]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:03.393 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-02-10T06:01:01.824606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48912,"kb_used_data":6096,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912304,"statfs":{"total":767960285184,"available":767910199296,"internally_reserved":0,"allocated":6242304,"data_stored":3488982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001934"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-02-10T05:59:49.892432+0000","last_change":"2024-02-10T05:59:35.863272+0000","last_active":"2024-02-10T05:59:49.892432+0000","last_peered":"2024-02-10T05:59:49.892432+0000","last_clean":"2024-02-10T05:59:49.892432+0000","last_became_active":"2024-02-10T05:59:35.862782+0000","last_became_peered":"2024-02-10T05:59:35.862782+0000","last_unstale":"2024-02-10T05:59:49.892432+0000","last_undegraded":"2024-02-10T05:59:49.892432+0000","last_fullsized":"2024-02-10T05:59:49.892432+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:41.599812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:41.599812+0000","last_clean_scrub_stamp":"2024-02-10T05:57:41.599812+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:52:42.267399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.63500000000000001}]},{"osd":1,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.56299999999999994}]},{"osd":3,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.66000000000000003}]},{"osd":4,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.37},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.49099999999999999}]},{"osd":5,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.41899999999999998}]},{"osd":6,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.51400000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.47499999999999998}]},{"osd":1,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.374},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.46300000000000002}]},{"osd":2,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.435}]},{"osd":3,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.316},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.48599999999999999}]},{"osd":7,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.499}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6328,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738824,"statfs":{"total":95995035648,"available":95988555776,"internally_reserved":0,"allocated":573440,"data_stored":237311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.32300000000000001,"15min":0.30299999999999999},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.314},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.35399999999999998,"15min":0.33600000000000002},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56100000000000005,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.42099999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.40200000000000002,"15min":0.40600000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55400000000000005,"5min":0.626,"15min":0.626},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38800000000000001,"15min":0.39200000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.495,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.32300000000000001}]},{"osd":3,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.441,"15min":0.441},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.379},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.441,"15min":0.441},"min":{"1min":0.34200000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.54700000000000004,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Feb 10 06:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76600000000000001,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56200000000000006,"15min":0.56699999999999995},"min":{"1min":0.42399999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.38700000000000001}]},{"osd":5,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.55600000000000005},"min":{"1min":0.44400000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.70699999999999996,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.70599999999999996,"5min":0.751,"15min":0.751},"last":0.45200000000000001}]},{"osd":6,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.51300000000000001}]},{"osd":7,"last update":"Sat Feb 10 06:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.439}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 06:00:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.316},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.48099999999999998,"5min":0.504,"15min":0.504},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.32800000000000001},"min":{"1min":0.25800000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.48999999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.25900000000000001}]},{"osd":2,"last update":"Sat Feb 10 06:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.36899999999999999,"15min":0.372},"min":{"1min":0.26300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.50700000000000001,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.41099999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.503,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.39900000000000002}]},{"osd":3,"last update":"Sat Feb 10 06:00:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.44900000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57399999999999995,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.443,"15min":0.442},"min":{"1min":0.34499999999999997,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.372}]},{"osd":4,"last update":"Sat Feb 10 06:00:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.83299999999999996,"15min":0.876},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80300000000000005,"5min":7.3040000000000003,"15min":7.3040000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.44600000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.755,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.41599999999999998}]},{"osd":5,"last update":"Sat Feb 10 06:00:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.48699999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.76900000000000002,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.60299999999999998}]},{"osd":6,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.44},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56999999999999995}]},{"osd":7,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.5},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.49399999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36599999999999999,"15min":0.36699999999999999},"min":{"1min":0.221,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53400000000000003,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.377},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.627,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.36599999999999999}]},{"osd":1,"last update":"Sat Feb 10 06:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41099999999999998,"15min":0.40899999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42199999999999999,"15min":0.42499999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.38}]},{"osd":3,"last update":"Sat Feb 10 06:00:09 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.45100000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.438,"15min":0.44},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56000000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Feb 10 06:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.67800000000000005}]},{"osd":5,"last update":"Sat Feb 10 06:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.625},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.68700000000000006}]},{"osd":6,"last update":"Sat Feb 10 06:00:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.64700000000000002}]},{"osd":7,"last update":"Sat Feb 10 06:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.63400000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.37,"15min":0.36899999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.41899999999999998,"15min":0.41599999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.59099999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.38300000000000001}]},{"osd":1,"last update":"Sat Feb 10 06:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.59599999999999997,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.36899999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.29799999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65000000000000002,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42999999999999999,"15min":0.432},"min":{"1min":0.314,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.57099999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.36299999999999999}]},{"osd":4,"last update":"Sat Feb 10 06:00:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60299999999999998,"15min":0.60699999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58799999999999997,"15min":0.59199999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.71999999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.44600000000000001}]},{"osd":5,"last update":"Sat Feb 10 06:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53200000000000003}]},{"osd":6,"last update":"Sat Feb 10 06:00:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.497},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.48199999999999998}]},{"osd":7,"last update":"Sat Feb 10 06:00:33 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.51500000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55700000000000005,"15min":0.56399999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69699999999999995,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.443},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.35599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.35699999999999998}]},{"osd":1,"last update":"Sat Feb 10 06:00:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72299999999999998,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72399999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Feb 10 06:00:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.75,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71099999999999997,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.432}]},{"osd":3,"last update":"Sat Feb 10 06:00:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.372,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Sat Feb 10 06:00:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.41799999999999998,"15min":0.40999999999999998},"min":{"1min":0.309,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.371},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.50700000000000001,"15min":0.49099999999999999},"min":{"1min":0.41799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.52400000000000002}]},{"osd":6,"last update":"Sat Feb 10 06:00:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.59299999999999997}]},{"osd":7,"last update":"Sat Feb 10 06:00:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.53900000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.71799999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.51400000000000001,"15min":0.50600000000000001},"min":{"1min":0.39600000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.73299999999999998,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.58499999999999996}]},{"osd":1,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.35899999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69699999999999995,"5min":0.749,"15min":0.749},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76000000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.42299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.54800000000000004},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.73999999999999999,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.39000000000000001}]},{"osd":3,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69899999999999995,"5min":0.746,"15min":0.746},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.53700000000000003},"min":{"1min":0.41899999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.503}]},{"osd":4,"last update":"Sat Feb 10 06:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.45400000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.47799999999999998,"15min":0.46600000000000003},"min":{"1min":0.36799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.46600000000000003}]},{"osd":6,"last update":"Sat Feb 10 06:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.54400000000000004}]},{"osd":7,"last update":"Sat Feb 10 06:00:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.47799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T06:01:03.395 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T06:01:03.395 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T06:01:03.395 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T06:01:03.395 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T06:01:03.610 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T06:01:04.072 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:03 smithi139 ceph-mon[24727]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:01:04.073 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:03 smithi139 ceph-mon[29453]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:01:04.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:03 smithi150 ceph-mon[25582]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:01:05.070 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:04 smithi139 ceph-mon[24727]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:05.070 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:04 smithi139 ceph-mon[29453]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:05.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:04 smithi150 ceph-mon[25582]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:05.380 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:01:05.380 INFO:teuthology.orchestra.run.smithi139.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T06:01:05.984 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:05 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/62846141' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T06:01:05.984 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:05 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/62846141' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T06:01:06.009 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T06:01:06.009 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T06:01:06.010 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:01:06.020 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi139.front.sepia.ceph.com 2024-02-10T06:01:06.020 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-02-10T06:01:06.335 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:05 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/62846141' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T06:01:07.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:06 smithi139 ceph-mon[29453]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:07.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:06 smithi139 ceph-mon[24727]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:07.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:06 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[47532]: level=info ts=2024-02-10T06:01:06.798Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002434363s 2024-02-10T06:01:07.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:06 smithi150 ceph-mon[25582]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:08.035 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:01:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:01:08.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:08 smithi139 ceph-mon[24727]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:08.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:08 smithi139 ceph-mon[24727]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T06:01:08.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:08 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2974296495' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T06:01:08.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:08 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T06:01:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:08 smithi139 ceph-mon[29453]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:08 smithi139 ceph-mon[29453]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T06:01:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:08 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2974296495' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T06:01:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:08 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T06:01:08.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:08 smithi150 ceph-mon[25582]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:08.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:08 smithi150 ceph-mon[25582]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T06:01:08.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:08 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2974296495' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T06:01:08.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:08 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T06:01:09.029 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:01:09.030 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028", 2024-02-10T06:01:09.030 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "r", 2024-02-10T06:01:09.030 INFO:teuthology.orchestra.run.smithi139.stdout: "current_period": "6175ce0e-7e28-4a50-b3b5-9fd14bdab1d0", 2024-02-10T06:01:09.030 INFO:teuthology.orchestra.run.smithi139.stdout: "epoch": 1 2024-02-10T06:01:09.030 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:01:09.124 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:08 smithi150 systemd[1]: Starting Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:01:09.408 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 podman[41688]: 2024-02-10 06:01:09.158003463 +0000 UTC m=+0.106632127 container create db9dee5d8d836c8eb027e50ecb585d7d1e337ed924fa3a672bd4104436ae8f09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, com.redhat.component=ubi8-container, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , release=236.1648460182, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, description=Ceph Grafana Container, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, vcs-type=git) 2024-02-10T06:01:09.408 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 podman[41688]: 2024-02-10 06:01:09.068581858 +0000 UTC m=+0.017210496 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-02-10T06:01:09.409 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 podman[41688]: 2024-02-10 06:01:09.360084771 +0000 UTC m=+0.308713395 container init db9dee5d8d836c8eb027e50ecb585d7d1e337ed924fa3a672bd4104436ae8f09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, vcs-type=git, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, architecture=x86_64, io.buildah.version=1.24.2, name=ubi8, maintainer=Paul Cuzner , vendor=Red Hat, Inc., distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-02-10T06:01:09.409 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 podman[41688]: 2024-02-10 06:01:09.362504187 +0000 UTC m=+0.311132819 container start db9dee5d8d836c8eb027e50ecb585d7d1e337ed924fa3a672bd4104436ae8f09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64) 2024-02-10T06:01:09.409 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-10T06:01:09.409 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="App mode production" logger=settings 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-10T06:01:09.410 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T06:01:09.411 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-02-10T06:01:09.412 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-02-10T06:01:09.413 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-02-10T06:01:09.413 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-02-10T06:01:09.413 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-02-10T06:01:09.413 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-02-10T06:01:09.413 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-02-10T06:01:09.413 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-10T06:01:09.413 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-02-10T06:01:09.413 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-02-10T06:01:09.660 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-02-10T06:01:09.661 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-02-10T06:01:09.662 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-02-10T06:01:09.663 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-02-10T06:01:09.663 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-02-10T06:01:09.663 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-02-10T06:01:09.663 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-02-10T06:01:09.663 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-02-10T06:01:09.663 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-02-10T06:01:09.663 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-02-10T06:01:09.663 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-02-10T06:01:09.663 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-02-10T06:01:09.663 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-02-10T06:01:09.664 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-02-10T06:01:09.664 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-02-10T06:01:09.664 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-02-10T06:01:09.664 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-02-10T06:01:09.664 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-02-10T06:01:09.664 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 bash[41688]: db9dee5d8d836c8eb027e50ecb585d7d1e337ed924fa3a672bd4104436ae8f09 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 systemd[1]: Started Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-02-10T06:01:09.665 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-10T06:01:09.666 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-10T06:01:09.667 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-10T06:01:09.668 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-10T06:01:09.669 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-10T06:01:09.670 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-10T06:01:09.671 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-10T06:01:09.671 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-10T06:01:09.671 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-10T06:01:09.671 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-10T06:01:09.671 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-02-10T06:01:09.671 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-10T06:01:09.671 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-02-10T06:01:09.671 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-02-10T06:01:09.671 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-10T06:01:09.671 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-10T06:01:09.671 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-10T06:01:09.748 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-10T06:01:09.911 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-10T06:01:09.912 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-10T06:01:09.912 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-10T06:01:09.912 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-10T06:01:09.912 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-02-10T06:01:09.912 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-02-10T06:01:09.912 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-02-10T06:01:09.912 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-02-10T06:01:09.913 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-10T06:01:09.913 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-10T06:01:09.913 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-10T06:01:09.913 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-10T06:01:09.913 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-02-10T06:01:09.914 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-10T06:01:09.915 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-10T06:01:09.916 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-10T06:01:09.917 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-10T06:01:09.918 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-02-10T06:01:09.919 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-02-10T06:01:09.920 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-02-10T06:01:09.920 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-02-10T06:01:09.920 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-10T06:01:09.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-10T06:01:09.922 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-02-10T06:01:09.923 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-02-10T06:01:09.923 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-10T06:01:09.923 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-02-10T06:01:09.923 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-10T06:01:09.923 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-10T06:01:09.923 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-10T06:01:09.923 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-02-10T06:01:10.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T06:01:10.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-mon[25582]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T06:01:10.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:10.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:10.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:10.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:10.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-mon[25582]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:10.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:10.277 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-02-10T06:01:10.277 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-02-10T06:01:10.277 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-02-10T06:01:10.278 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-02-10T06:01:10.278 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-10T06:01:10.278 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-02-10T06:01:10.278 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-02-10T06:01:10.278 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-02-10T06:01:10.278 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-02-10T06:01:10.278 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-02-10T06:01:10.279 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-02-10T06:01:10.279 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-02-10T06:01:10.279 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-02-10T06:01:10.279 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-02-10T06:01:10.279 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-02-10T06:01:10.279 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-02-10T06:01:10.280 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-02-10T06:01:10.280 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-02-10T06:01:10.280 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-02-10T06:01:10.280 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-02-10T06:01:10.280 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-02-10T06:01:10.280 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-02-10T06:01:10.281 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=564.859364ms 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-10T06:01:10.282 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:09+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-10T06:01:10.283 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:10+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-10T06:01:10.283 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:10+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-10T06:01:10.283 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:10+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-10T06:01:10.283 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:10+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-10T06:01:10.283 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:10+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-10T06:01:10.283 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:10+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-10T06:01:10.283 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:10+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T06:01:10.283 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:01:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:01:10+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-10T06:01:10.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T06:01:10.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[24727]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T06:01:10.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:10.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:10.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:10.295 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:10.295 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[24727]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:10.295 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:10.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T06:01:10.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[29453]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T06:01:10.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:10.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:10.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:10.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:10.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[29453]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:10.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:11.002 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:01:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:01:10] "GET /metrics HTTP/1.1" 200 191521 "" "Prometheus/2.33.4" 2024-02-10T06:01:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:11 smithi139 ceph-mon[24727]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T06:01:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:11 smithi139 ceph-mon[29453]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T06:01:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:11 smithi150 ceph-mon[25582]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T06:01:12.087 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:01:12.087 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "84401f0e-29b9-42d6-8b93-fc8880fde592", 2024-02-10T06:01:12.087 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default", 2024-02-10T06:01:12.087 INFO:teuthology.orchestra.run.smithi139.stdout: "api_name": "default", 2024-02-10T06:01:12.087 INFO:teuthology.orchestra.run.smithi139.stdout: "is_master": "true", 2024-02-10T06:01:12.087 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [], 2024-02-10T06:01:12.087 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames": [], 2024-02-10T06:01:12.087 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames_s3website": [], 2024-02-10T06:01:12.087 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "", 2024-02-10T06:01:12.087 INFO:teuthology.orchestra.run.smithi139.stdout: "zones": [], 2024-02-10T06:01:12.087 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_targets": [], 2024-02-10T06:01:12.088 INFO:teuthology.orchestra.run.smithi139.stdout: "default_placement": "", 2024-02-10T06:01:12.088 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028", 2024-02-10T06:01:12.088 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_policy": { 2024-02-10T06:01:12.088 INFO:teuthology.orchestra.run.smithi139.stdout: "groups": [] 2024-02-10T06:01:12.088 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:01:12.088 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:01:12.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:12 smithi139 ceph-mon[24727]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T06:01:12.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:12 smithi139 ceph-mon[29453]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T06:01:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:12 smithi150 ceph-mon[25582]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T06:01:12.775 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-02-10T06:01:14.030 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:13 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:14.030 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:13 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:14.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:13 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:14.966 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:14 smithi139 ceph-mon[24727]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T06:01:14.967 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:14 smithi139 ceph-mon[29453]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T06:01:15.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:14 smithi150 ceph-mon[25582]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T06:01:15.848 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:01:15.848 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "8a9439ca-55d9-426a-935b-f5ca0d651aa4", 2024-02-10T06:01:15.848 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "z", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "z.rgw.meta:root", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "z.rgw.control", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "z.rgw.log:gc", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "z.rgw.log:lc", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "z.rgw.log", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "z.rgw.meta:roles", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "z.rgw.otp", 2024-02-10T06:01:15.849 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "", 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "" 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "z.rgw.buckets.index", 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "z.rgw.buckets.data" 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0 2024-02-10T06:01:15.850 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:01:15.851 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:01:15.851 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-02-10T06:01:15.851 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028", 2024-02-10T06:01:15.851 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "z.rgw.log:notif" 2024-02-10T06:01:15.851 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:01:16.502 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:16 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:16.502 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:16 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:16.502 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:16 smithi139 ceph-mon[24727]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T06:01:16.502 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:16 smithi139 ceph-mon[24727]: Reconfiguring daemon alertmanager.a on smithi139 2024-02-10T06:01:16.502 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:16 smithi139 ceph-mon[24727]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:01:16.503 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:16 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:16.503 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:16 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:16.503 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:16 smithi139 ceph-mon[29453]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T06:01:16.503 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:16 smithi139 ceph-mon[29453]: Reconfiguring daemon alertmanager.a on smithi139 2024-02-10T06:01:16.503 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:16 smithi139 ceph-mon[29453]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:01:16.525 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-02-10T06:01:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:16 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:16 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:16 smithi150 ceph-mon[25582]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T06:01:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:16 smithi150 ceph-mon[25582]: Reconfiguring daemon alertmanager.a on smithi139 2024-02-10T06:01:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:16 smithi150 ceph-mon[25582]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:01:18.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:01:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:01:19.288 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:18 smithi139 ceph-mon[24727]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 811 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-02-10T06:01:19.288 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:18 smithi139 ceph-mon[29453]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 811 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-02-10T06:01:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:18 smithi150 ceph-mon[25582]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 811 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-02-10T06:01:20.336 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:20 smithi139 systemd[1]: Stopping Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:01:20.886 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:01:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:01:20] "GET /metrics HTTP/1.1" 200 193186 "" "Prometheus/2.33.4" 2024-02-10T06:01:20.886 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:20 smithi139 bash[49239]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T06:01:20.886 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[47532]: level=info ts=2024-02-10T06:01:20.797Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:01:20.886 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:20 smithi139 podman[49313]: 2024-02-10 06:01:20.81813398 +0000 UTC m=+0.125880327 container died 623fed35712f77b9c7c9ea7f6feebd0be9f84ea9bf00892565d5f85d0cb2e7ec (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:21.264 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:20 smithi139 ceph-mon[24727]: pgmap v51: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-02-10T06:01:21.265 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:20 smithi139 ceph-mon[29453]: pgmap v51: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-02-10T06:01:21.308 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:20 smithi150 ceph-mon[25582]: pgmap v51: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-02-10T06:01:21.532 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:21 smithi139 podman[49313]: 2024-02-10 06:01:21.265049251 +0000 UTC m=+0.572795590 container cleanup 623fed35712f77b9c7c9ea7f6feebd0be9f84ea9bf00892565d5f85d0cb2e7ec (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:21.533 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:21 smithi139 bash[49313]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T06:01:21.533 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:21 smithi139 bash[49340]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T06:01:21.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:21 smithi139 podman[49326]: 2024-02-10 06:01:21.532852335 +0000 UTC m=+0.714915733 container remove 623fed35712f77b9c7c9ea7f6feebd0be9f84ea9bf00892565d5f85d0cb2e7ec (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:21.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:21 smithi139 podman[49326]: 2024-02-10 06:01:21.599971065 +0000 UTC m=+0.782034427 volume remove be60df45ef03aa043b4c420b2d7c20ceb6076314147e191787986d7fdb389b73 2024-02-10T06:01:21.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:21 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T06:01:21.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:21 smithi139 systemd[1]: Stopped Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:01:21.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:21 smithi139 systemd[1]: Starting Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:01:22.167 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:21 smithi139 ceph-mon[24727]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T06:01:22.167 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:21 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T06:01:22.167 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:21 smithi139 ceph-mon[29453]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T06:01:22.168 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:21 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T06:01:22.168 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:21 smithi139 podman[49412]: 2024-02-10 06:01:21.739879917 +0000 UTC m=+0.018740600 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-10T06:01:22.168 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:21 smithi139 podman[49412]: 2024-02-10 06:01:21.859108024 +0000 UTC m=+0.137968735 volume create bd138c9d3208bb8540ff7bdcc3812ac52c0c00fcc5ac197b974c1604ab3b0eb6 2024-02-10T06:01:22.168 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:21 smithi139 podman[49412]: 2024-02-10 06:01:21.967941469 +0000 UTC m=+0.246802138 container create adb2e2c020a12217ed58259c8a77e36f60ded6946a537ce4f89888a99901d534 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:21 smithi150 ceph-mon[25582]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T06:01:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:21 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T06:01:22.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 podman[49412]: 2024-02-10 06:01:22.270627867 +0000 UTC m=+0.549488536 container init adb2e2c020a12217ed58259c8a77e36f60ded6946a537ce4f89888a99901d534 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:22.587 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 podman[49412]: 2024-02-10 06:01:22.273215733 +0000 UTC m=+0.552076403 container start adb2e2c020a12217ed58259c8a77e36f60ded6946a537ce4f89888a99901d534 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:22.587 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=info ts=2024-02-10T06:01:22.283Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-10T06:01:22.587 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=info ts=2024-02-10T06:01:22.283Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-10T06:01:22.587 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=info ts=2024-02-10T06:01:22.284Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.139 port=9094 2024-02-10T06:01:22.587 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=info ts=2024-02-10T06:01:22.285Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T06:01:22.587 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=info ts=2024-02-10T06:01:22.310Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T06:01:22.587 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=info ts=2024-02-10T06:01:22.310Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T06:01:22.587 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=info ts=2024-02-10T06:01:22.312Z caller=main.go:518 msg=Listening address=:9093 2024-02-10T06:01:22.587 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=info ts=2024-02-10T06:01:22.312Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-10T06:01:22.587 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 bash[49412]: adb2e2c020a12217ed58259c8a77e36f60ded6946a537ce4f89888a99901d534 2024-02-10T06:01:22.587 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:22 smithi139 systemd[1]: Started Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:01:23.067 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:22 smithi150 ceph-mon[25582]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 819 B/s wr, 3 op/s 2024-02-10T06:01:23.067 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:22 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-10T06:01:23.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:22 smithi150 ceph-mon[25582]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T06:01:23.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:22 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:23.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-mon[24727]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 819 B/s wr, 3 op/s 2024-02-10T06:01:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-10T06:01:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-mon[24727]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T06:01:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-mon[29453]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 819 B/s wr, 3 op/s 2024-02-10T06:01:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-10T06:01:23.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-mon[29453]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T06:01:23.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:22 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:24.151 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:23 smithi150 ceph-mon[25582]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:01:24.152 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:23 smithi150 ceph-mon[25582]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:01:24.152 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:23 smithi150 ceph-mon[25582]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T06:01:24.152 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:23 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T06:01:24.285 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:23 smithi139 ceph-mon[29453]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:01:24.285 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:23 smithi139 ceph-mon[29453]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:01:24.285 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:23 smithi139 ceph-mon[29453]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T06:01:24.285 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:23 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T06:01:24.286 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:23 smithi139 ceph-mon[24727]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:01:24.286 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:23 smithi139 ceph-mon[24727]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:01:24.286 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:23 smithi139 ceph-mon[24727]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T06:01:24.286 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:23 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T06:01:24.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:24 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=info ts=2024-02-10T06:01:24.285Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000323421s 2024-02-10T06:01:25.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:24 smithi139 ceph-mon[24727]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-02-10T06:01:25.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:24 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-10T06:01:25.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:24 smithi139 ceph-mon[24727]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T06:01:25.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:24 smithi139 ceph-mon[29453]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-02-10T06:01:25.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:24 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-10T06:01:25.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:24 smithi139 ceph-mon[29453]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T06:01:25.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:24 smithi150 ceph-mon[25582]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-02-10T06:01:25.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:24 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3200915059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-10T06:01:25.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:24 smithi150 ceph-mon[25582]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T06:01:25.663 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 systemd[1]: Stopping Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:01:25.663 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 bash[42381]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T06:01:25.927 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.663Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:01:25.927 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.663Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T06:01:25.927 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.663Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T06:01:25.927 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.663Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T06:01:25.927 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.663Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T06:01:25.927 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.663Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T06:01:25.927 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.664Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T06:01:25.928 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.664Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T06:01:25.928 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.664Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T06:01:25.928 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.664Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T06:01:25.928 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.664Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T06:01:25.928 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[40702]: ts=2024-02-10T06:01:25.664Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T06:01:25.928 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:25 smithi150 podman[42394]: 2024-02-10 06:01:25.673634599 +0000 UTC m=+0.120685071 container died 00092498742fd21aec3e02c8330aec295c8c74032b293a8f5a881e0665385ba9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:26.221 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:25 smithi139 ceph-mon[24727]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T06:01:26.221 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:25 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T06:01:26.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:25 smithi139 ceph-mon[29453]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T06:01:26.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:25 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T06:01:26.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-mon[25582]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T06:01:26.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:25 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T06:01:26.374 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 podman[42394]: 2024-02-10 06:01:26.121774171 +0000 UTC m=+0.568824707 container cleanup 00092498742fd21aec3e02c8330aec295c8c74032b293a8f5a881e0665385ba9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:26.374 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 bash[42394]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T06:01:26.374 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 bash[42421]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T06:01:26.807 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 podman[42408]: 2024-02-10 06:01:26.473075512 +0000 UTC m=+0.801224826 container remove 00092498742fd21aec3e02c8330aec295c8c74032b293a8f5a881e0665385ba9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:26.807 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-10T06:01:26.807 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 systemd[1]: Stopped Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:01:26.808 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 systemd[1]: Starting Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:01:26.808 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 podman[42492]: 2024-02-10 06:01:26.606167297 +0000 UTC m=+0.019946182 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-10T06:01:26.808 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 podman[42492]: 2024-02-10 06:01:26.707265513 +0000 UTC m=+0.121044395 container create 81a684e8f18fa525dffca8ce2161701f4b6cc91d6dc0bddeeb1d6e15950ab24b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:27.152 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-mon[25582]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:01:27.152 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-10T06:01:27.152 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-mon[25582]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T06:01:27.152 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T06:01:27.153 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 podman[42492]: 2024-02-10 06:01:26.901041447 +0000 UTC m=+0.314820332 container init 81a684e8f18fa525dffca8ce2161701f4b6cc91d6dc0bddeeb1d6e15950ab24b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:27.153 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 podman[42492]: 2024-02-10 06:01:26.903424901 +0000 UTC m=+0.317203792 container start 81a684e8f18fa525dffca8ce2161701f4b6cc91d6dc0bddeeb1d6e15950ab24b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:01:27.153 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.927Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-10T06:01:27.153 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.927Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-10T06:01:27.153 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.927Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-10T06:01:27.153 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.927Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi150 (none))" 2024-02-10T06:01:27.153 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.927Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T06:01:27.153 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.927Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T06:01:27.153 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.928Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T06:01:27.153 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.929Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-10T06:01:27.153 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.930Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-10T06:01:27.154 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.933Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T06:01:27.154 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.933Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.884µs 2024-02-10T06:01:27.154 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:26.933Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T06:01:27.154 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 bash[42492]: 81a684e8f18fa525dffca8ce2161701f4b6cc91d6dc0bddeeb1d6e15950ab24b 2024-02-10T06:01:27.154 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:26 smithi150 systemd[1]: Started Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:01:27.154 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:27.150Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-02-10T06:01:27.154 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:27.150Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-02-10T06:01:27.154 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:27.150Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=43.952µs wal_replay_duration=216.926518ms total_replay_duration=216.990974ms 2024-02-10T06:01:27.154 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:27.152Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T06:01:27.154 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:27.152Z caller=main.go:947 level=info msg="TSDB started" 2024-02-10T06:01:27.154 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:27.152Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T06:01:27.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:26 smithi139 ceph-mon[29453]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:01:27.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:26 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-10T06:01:27.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:26 smithi139 ceph-mon[29453]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T06:01:27.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:26 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T06:01:27.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:26 smithi139 ceph-mon[24727]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:01:27.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:26 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-10T06:01:27.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:26 smithi139 ceph-mon[24727]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T06:01:27.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:26 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T06:01:27.613 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:27.172Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.366914ms db_storage=801ns remote_storage=1.64µs web_handler=468ns query_engine=848ns scrape=510.3µs scrape_sd=37.703µs notify=31.549µs notify_sd=12.779µs rules=18.461746ms 2024-02-10T06:01:27.613 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:01:27.172Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-10T06:01:27.982 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:01:27.982 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "335ad1e8-d466-40e0-95c2-09f32e101058", 2024-02-10T06:01:27.982 INFO:teuthology.orchestra.run.smithi139.stdout: "epoch": 1, 2024-02-10T06:01:27.982 INFO:teuthology.orchestra.run.smithi139.stdout: "predecessor_uuid": "6175ce0e-7e28-4a50-b3b5-9fd14bdab1d0", 2024-02-10T06:01:27.982 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": [], 2024-02-10T06:01:27.982 INFO:teuthology.orchestra.run.smithi139.stdout: "period_map": { 2024-02-10T06:01:27.982 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "335ad1e8-d466-40e0-95c2-09f32e101058", 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "zonegroups": [ 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "84401f0e-29b9-42d6-8b93-fc8880fde592", 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default", 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "api_name": "default", 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "is_master": "true", 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [], 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames": [], 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames_s3website": [], 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "8a9439ca-55d9-426a-935b-f5ca0d651aa4", 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "zones": [ 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "8a9439ca-55d9-426a-935b-f5ca0d651aa4", 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "z", 2024-02-10T06:01:27.983 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [], 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": "false", 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": "false", 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": "false", 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": "true", 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "" 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_targets": [ 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default-placement", 2024-02-10T06:01:27.984 INFO:teuthology.orchestra.run.smithi139.stdout: "tags": [], 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": [ 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD" 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: "default_placement": "default-placement", 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028", 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_policy": { 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: "groups": [] 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: "short_zone_ids": [ 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "8a9439ca-55d9-426a-935b-f5ca0d651aa4", 2024-02-10T06:01:27.985 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 103908705 2024-02-10T06:01:27.986 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:01:27.986 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-02-10T06:01:27.986 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:01:27.986 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zonegroup": "84401f0e-29b9-42d6-8b93-fc8880fde592", 2024-02-10T06:01:27.986 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "8a9439ca-55d9-426a-935b-f5ca0d651aa4", 2024-02-10T06:01:27.986 INFO:teuthology.orchestra.run.smithi139.stdout: "period_config": { 2024-02-10T06:01:27.986 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_quota": { 2024-02-10T06:01:27.986 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2024-02-10T06:01:27.986 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2024-02-10T06:01:27.986 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2024-02-10T06:01:27.986 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2024-02-10T06:01:27.987 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2024-02-10T06:01:27.987 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:01:27.987 INFO:teuthology.orchestra.run.smithi139.stdout: "user_quota": { 2024-02-10T06:01:27.987 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2024-02-10T06:01:27.987 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2024-02-10T06:01:27.987 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2024-02-10T06:01:27.987 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2024-02-10T06:01:27.987 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2024-02-10T06:01:27.987 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:01:27.987 INFO:teuthology.orchestra.run.smithi139.stdout: "user_ratelimit": { 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_ops": 0, 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_ops": 0, 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_bytes": 0, 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_bytes": 0, 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_ratelimit": { 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_ops": 0, 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_ops": 0, 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_bytes": 0, 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_bytes": 0, 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:01:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: "anonymous_ratelimit": { 2024-02-10T06:01:27.989 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_ops": 0, 2024-02-10T06:01:27.989 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_ops": 0, 2024-02-10T06:01:27.989 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_bytes": 0, 2024-02-10T06:01:27.989 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_bytes": 0, 2024-02-10T06:01:27.989 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false 2024-02-10T06:01:27.989 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:01:27.989 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:01:27.989 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028", 2024-02-10T06:01:27.989 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_name": "r", 2024-02-10T06:01:27.989 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2024-02-10T06:01:27.989 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.139:9093"}]: dispatch 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.150:9095"}]: dispatch 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.150:3000"}]: dispatch 2024-02-10T06:01:28.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[24727]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.139:9093"}]: dispatch 2024-02-10T06:01:28.295 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.296 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:01:28.296 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.150:9095"}]: dispatch 2024-02-10T06:01:28.296 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.296 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:01:28.296 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.150:3000"}]: dispatch 2024-02-10T06:01:28.296 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.296 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:28.296 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:28.296 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:28.296 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-10T06:01:28.296 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:27 smithi139 ceph-mon[29453]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.139:9093"}]: dispatch 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.150:9095"}]: dispatch 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.150:3000"}]: dispatch 2024-02-10T06:01:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:28.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:28.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:28.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:28.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3211605681' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-10T06:01:28.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:27 smithi150 ceph-mon[25582]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T06:01:28.752 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-02-10T06:01:29.083 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:01:29.083 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.139:9093"}]: dispatch 2024-02-10T06:01:29.083 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:01:29.083 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.150:9095"}]: dispatch 2024-02-10T06:01:29.083 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:01:29.084 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.150:3000"}]: dispatch 2024-02-10T06:01:29.084 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[24727]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:01:29.084 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:01:29.084 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.139:9093"}]: dispatch 2024-02-10T06:01:29.084 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:01:29.084 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.150:9095"}]: dispatch 2024-02-10T06:01:29.084 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:01:29.084 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.150:3000"}]: dispatch 2024-02-10T06:01:29.084 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:28 smithi139 ceph-mon[29453]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:01:29.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:28 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:01:29.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:28 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.139:9093"}]: dispatch 2024-02-10T06:01:29.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:28 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:01:29.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:28 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.150:9095"}]: dispatch 2024-02-10T06:01:29.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:28 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:01:29.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:28 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.150:3000"}]: dispatch 2024-02-10T06:01:29.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:28 smithi150 ceph-mon[25582]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:01:31.115 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled rgw.foo update... 2024-02-10T06:01:31.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:30 smithi150 ceph-mon[25582]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 208 B/s rd, 417 B/s wr, 1 op/s 2024-02-10T06:01:31.307 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:30 smithi139 ceph-mon[24727]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 208 B/s rd, 417 B/s wr, 1 op/s 2024-02-10T06:01:31.307 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:30 smithi139 ceph-mon[29453]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 208 B/s rd, 417 B/s wr, 1 op/s 2024-02-10T06:01:31.962 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph osd pool create foo' 2024-02-10T06:01:32.215 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:32 smithi139 ceph-mon[24727]: from='client.24580 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:01:32.215 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:32 smithi139 ceph-mon[24727]: Saving service rgw.foo spec with placement count:2 2024-02-10T06:01:32.215 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:32 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:32.215 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:32 smithi139 ceph-mon[24727]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:01:32.215 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:32 smithi139 ceph-mon[29453]: from='client.24580 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:01:32.215 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:32 smithi139 ceph-mon[29453]: Saving service rgw.foo spec with placement count:2 2024-02-10T06:01:32.215 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:32 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:32.215 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:32 smithi139 ceph-mon[29453]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:01:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:32 smithi150 ceph-mon[25582]: from='client.24580 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:01:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:32 smithi150 ceph-mon[25582]: Saving service rgw.foo spec with placement count:2 2024-02-10T06:01:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:32 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:32 smithi150 ceph-mon[25582]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:01:32.558 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:01:32 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=info ts=2024-02-10T06:01:32.286Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001388812s 2024-02-10T06:01:33.812 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.812 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.812 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.812 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.813 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.813 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: Saving service rgw.foo spec with placement count:2 2024-02-10T06:01:33.813 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.813 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:33.813 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:33.813 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T06:01:33.813 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.813 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:33.813 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[29453]: Deploying daemon rgw.foo.smithi139.kpyuou on smithi139 2024-02-10T06:01:33.813 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.813 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.814 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.814 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.814 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.814 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: Saving service rgw.foo spec with placement count:2 2024-02-10T06:01:33.814 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.814 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:33.814 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:33.814 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T06:01:33.814 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.814 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:33.814 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:33 smithi139 ceph-mon[24727]: Deploying daemon rgw.foo.smithi139.kpyuou on smithi139 2024-02-10T06:01:33.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: Saving service rgw.foo spec with placement count:2 2024-02-10T06:01:33.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:33.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:33.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T06:01:33.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:33.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:33.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:33 smithi150 ceph-mon[25582]: Deploying daemon rgw.foo.smithi139.kpyuou on smithi139 2024-02-10T06:01:35.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:34 smithi139 ceph-mon[24727]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 907 B/s wr, 2 op/s 2024-02-10T06:01:35.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:34 smithi139 ceph-mon[29453]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 907 B/s wr, 2 op/s 2024-02-10T06:01:35.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:34 smithi150 ceph-mon[25582]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 907 B/s wr, 2 op/s 2024-02-10T06:01:35.847 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'foo' created 2024-02-10T06:01:36.023 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:35 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/4183194135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T06:01:36.023 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:35 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T06:01:36.023 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:35 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/4183194135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T06:01:36.024 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:35 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T06:01:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:35 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/4183194135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T06:01:36.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:35 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T06:01:36.520 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'rbd pool init foo' 2024-02-10T06:01:37.082 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:36 smithi139 ceph-mon[24727]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 805 B/s wr, 2 op/s 2024-02-10T06:01:37.083 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:36 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-10T06:01:37.083 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:36 smithi139 ceph-mon[24727]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T06:01:37.083 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:36 smithi139 ceph-mon[29453]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 805 B/s wr, 2 op/s 2024-02-10T06:01:37.083 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:36 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-10T06:01:37.083 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:36 smithi139 ceph-mon[29453]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T06:01:37.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:36 smithi150 ceph-mon[25582]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 805 B/s wr, 2 op/s 2024-02-10T06:01:37.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:36 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-10T06:01:37.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:36 smithi150 ceph-mon[25582]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T06:01:37.854 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:01:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:01:38.122 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[24727]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T06:01:38.122 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:38.122 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:38.122 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:38.122 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T06:01:38.122 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:38.122 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:38.122 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[24727]: Deploying daemon rgw.foo.smithi150.subshb on smithi150 2024-02-10T06:01:38.123 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[29453]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T06:01:38.123 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:38.123 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:38.123 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:38.123 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T06:01:38.123 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:38.123 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:38.123 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:37 smithi139 ceph-mon[29453]: Deploying daemon rgw.foo.smithi150.subshb on smithi150 2024-02-10T06:01:38.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:37 smithi150 ceph-mon[25582]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T06:01:38.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:37 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:38.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:37 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:38.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:37 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:01:38.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:37 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T06:01:38.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:37 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:38.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:37 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:38.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:37 smithi150 ceph-mon[25582]: Deploying daemon rgw.foo.smithi150.subshb on smithi150 2024-02-10T06:01:39.036 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:38 smithi150 ceph-mon[25582]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-02-10T06:01:39.036 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:38 smithi150 ceph-mon[25582]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T06:01:39.036 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:38 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3693982041' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-10T06:01:39.037 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:38 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-10T06:01:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:38 smithi139 ceph-mon[24727]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-02-10T06:01:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:38 smithi139 ceph-mon[24727]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T06:01:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:38 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3693982041' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-10T06:01:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:38 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-10T06:01:39.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:38 smithi139 ceph-mon[29453]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-02-10T06:01:39.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:38 smithi139 ceph-mon[29453]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T06:01:39.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:38 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3693982041' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-10T06:01:39.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:38 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-10T06:01:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-10T06:01:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[24727]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T06:01:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:40.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-10T06:01:40.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[29453]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T06:01:40.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:40.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:40.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:40.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:39 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:39 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-10T06:01:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:39 smithi150 ceph-mon[25582]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T06:01:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:39 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:39 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:39 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:39 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:40.813 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:01:40 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:01:40] "GET /metrics HTTP/1.1" 200 196865 "" "Prometheus/2.33.4" 2024-02-10T06:01:41.084 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:40 smithi150 ceph-mon[25582]: pgmap v73: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:41.084 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:40 smithi150 ceph-mon[25582]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T06:01:41.084 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:40 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:41.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:40 smithi139 ceph-mon[24727]: pgmap v73: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:41.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:40 smithi139 ceph-mon[24727]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T06:01:41.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:40 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:41.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:40 smithi139 ceph-mon[29453]: pgmap v73: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:41.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:40 smithi139 ceph-mon[29453]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T06:01:41.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:40 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:42.016 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch apply iscsi foo u p' 2024-02-10T06:01:42.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:41 smithi150 ceph-mon[25582]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T06:01:42.237 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:41 smithi139 ceph-mon[24727]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T06:01:42.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:41 smithi139 ceph-mon[29453]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T06:01:43.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:42 smithi150 ceph-mon[25582]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 278 KiB/s rd, 8.7 KiB/s wr, 530 op/s 2024-02-10T06:01:43.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:42 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:43.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:42 smithi139 ceph-mon[24727]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 278 KiB/s rd, 8.7 KiB/s wr, 530 op/s 2024-02-10T06:01:43.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:42 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:43.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:42 smithi139 ceph-mon[29453]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 278 KiB/s rd, 8.7 KiB/s wr, 530 op/s 2024-02-10T06:01:43.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:42 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.064 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.065 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[24727]: Checking dashboard <-> RGW credentials 2024-02-10T06:01:44.065 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.065 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.065 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.065 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:44.065 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:44.065 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:44.065 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.065 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[29453]: Checking dashboard <-> RGW credentials 2024-02-10T06:01:44.065 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.066 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.066 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.066 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:44.066 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:44.066 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:43 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:44.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:43 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:43 smithi150 ceph-mon[25582]: Checking dashboard <-> RGW credentials 2024-02-10T06:01:44.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:43 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:43 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:43 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:44.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:43 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:44.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:43 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:44.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:43 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:44.489 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled iscsi.foo update... 2024-02-10T06:01:45.060 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:44 smithi139 ceph-mon[24727]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 186 KiB/s rd, 5.9 KiB/s wr, 355 op/s 2024-02-10T06:01:45.060 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:44 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:45.060 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:44 smithi139 ceph-mon[29453]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 186 KiB/s rd, 5.9 KiB/s wr, 355 op/s 2024-02-10T06:01:45.060 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:44 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:45.083 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'sleep 180' 2024-02-10T06:01:45.325 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:44 smithi150 ceph-mon[25582]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 186 KiB/s rd, 5.9 KiB/s wr, 355 op/s 2024-02-10T06:01:45.325 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:44 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:46.253 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:45 smithi139 ceph-mon[24727]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:01:46.253 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:45 smithi139 ceph-mon[24727]: Saving service iscsi.foo spec with placement count:1 2024-02-10T06:01:46.254 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:45 smithi139 ceph-mon[29453]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:01:46.254 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:45 smithi139 ceph-mon[29453]: Saving service iscsi.foo spec with placement count:1 2024-02-10T06:01:46.345 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:45 smithi150 ceph-mon[25582]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:01:46.345 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:45 smithi150 ceph-mon[25582]: Saving service iscsi.foo spec with placement count:1 2024-02-10T06:01:47.236 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[24727]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 5.5 KiB/s wr, 312 op/s 2024-02-10T06:01:47.237 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:47.237 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:47.237 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:01:47.237 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:01:47.237 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T06:01:47.237 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:47.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[29453]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 5.5 KiB/s wr, 312 op/s 2024-02-10T06:01:47.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:47.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:47.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:01:47.238 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:01:47.238 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T06:01:47.238 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:46 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:46 smithi150 ceph-mon[25582]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 5.5 KiB/s wr, 312 op/s 2024-02-10T06:01:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:46 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:46 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:46 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:01:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:46 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:01:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:46 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T06:01:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:46 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:48.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:01:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:01:48.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:47 smithi150 ceph-mon[25582]: Checking dashboard <-> RGW credentials 2024-02-10T06:01:48.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:47 smithi150 ceph-mon[25582]: Checking pool "foo" exists for service iscsi.foo 2024-02-10T06:01:48.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:47 smithi150 ceph-mon[25582]: Deploying daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:01:48.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:47 smithi139 ceph-mon[24727]: Checking dashboard <-> RGW credentials 2024-02-10T06:01:48.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:47 smithi139 ceph-mon[24727]: Checking pool "foo" exists for service iscsi.foo 2024-02-10T06:01:48.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:47 smithi139 ceph-mon[24727]: Deploying daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:01:48.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:47 smithi139 ceph-mon[29453]: Checking dashboard <-> RGW credentials 2024-02-10T06:01:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:47 smithi139 ceph-mon[29453]: Checking pool "foo" exists for service iscsi.foo 2024-02-10T06:01:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:47 smithi139 ceph-mon[29453]: Deploying daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:01:49.037 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:48 smithi139 ceph-mon[24727]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 4.7 KiB/s wr, 271 op/s 2024-02-10T06:01:49.037 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:48 smithi139 ceph-mon[24727]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T06:01:49.038 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:48 smithi139 ceph-mon[29453]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 4.7 KiB/s wr, 271 op/s 2024-02-10T06:01:49.038 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:48 smithi139 ceph-mon[29453]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T06:01:49.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:48 smithi150 ceph-mon[25582]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 4.7 KiB/s wr, 271 op/s 2024-02-10T06:01:49.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:48 smithi150 ceph-mon[25582]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T06:01:50.270 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: mgrmap e20: y(active, since 119s), standbys: x 2024-02-10T06:01:50.270 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:50.270 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:01:50.271 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:01:50.272 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:50.272 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: mgrmap e20: y(active, since 119s), standbys: x 2024-02-10T06:01:50.272 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:50.272 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:50.272 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:50.272 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:50.272 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T06:01:50.272 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T06:01:50.272 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T06:01:50.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T06:01:50.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:01:50.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:01:50.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:01:50.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:01:50.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: mgrmap e20: y(active, since 119s), standbys: x 2024-02-10T06:01:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:01:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:50.985 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:01:50] "GET /metrics HTTP/1.1" 200 208042 "" "Prometheus/2.33.4" 2024-02-10T06:01:50.985 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[24727]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 343 B/s wr, 5 op/s 2024-02-10T06:01:50.985 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1939900846' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:01:50.985 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1469493957' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2159038905"}]: dispatch 2024-02-10T06:01:50.985 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2159038905"}]: dispatch 2024-02-10T06:01:50.986 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-02-10T06:01:50.986 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-10T06:01:50.986 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-10T06:01:50.986 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-02-10T06:01:50.986 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2159038905"}]': finished 2024-02-10T06:01:50.986 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[24727]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T06:01:51.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[29453]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 343 B/s wr, 5 op/s 2024-02-10T06:01:51.263 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1939900846' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:01:51.263 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1469493957' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2159038905"}]: dispatch 2024-02-10T06:01:51.263 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2159038905"}]: dispatch 2024-02-10T06:01:51.263 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-02-10T06:01:51.263 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-10T06:01:51.263 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-10T06:01:51.263 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-02-10T06:01:51.263 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2159038905"}]': finished 2024-02-10T06:01:51.263 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:50 smithi139 ceph-mon[29453]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T06:01:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:50 smithi150 ceph-mon[25582]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 343 B/s wr, 5 op/s 2024-02-10T06:01:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:50 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1939900846' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:01:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:50 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1469493957' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2159038905"}]: dispatch 2024-02-10T06:01:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:50 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2159038905"}]: dispatch 2024-02-10T06:01:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-02-10T06:01:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-10T06:01:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-10T06:01:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-02-10T06:01:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:50 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2159038905"}]': finished 2024-02-10T06:01:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:50 smithi150 ceph-mon[25582]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T06:01:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:51 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2362844063' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/3587488312"}]: dispatch 2024-02-10T06:01:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:51 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2362844063' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/3587488312"}]': finished 2024-02-10T06:01:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:51 smithi139 ceph-mon[24727]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T06:01:52.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:51 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2362844063' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/3587488312"}]: dispatch 2024-02-10T06:01:52.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:51 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2362844063' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/3587488312"}]': finished 2024-02-10T06:01:52.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:51 smithi139 ceph-mon[29453]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T06:01:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:51 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2362844063' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/3587488312"}]: dispatch 2024-02-10T06:01:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:51 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2362844063' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/3587488312"}]': finished 2024-02-10T06:01:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:51 smithi150 ceph-mon[25582]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T06:01:53.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:52 smithi139 ceph-mon[29453]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-02-10T06:01:53.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:52 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2203703832' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/968128567"}]: dispatch 2024-02-10T06:01:53.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:52 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/968128567"}]: dispatch 2024-02-10T06:01:53.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:52 smithi139 ceph-mon[24727]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-02-10T06:01:53.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:52 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2203703832' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/968128567"}]: dispatch 2024-02-10T06:01:53.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:52 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/968128567"}]: dispatch 2024-02-10T06:01:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:52 smithi150 ceph-mon[25582]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-02-10T06:01:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:52 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2203703832' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/968128567"}]: dispatch 2024-02-10T06:01:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:52 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/968128567"}]: dispatch 2024-02-10T06:01:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/968128567"}]': finished 2024-02-10T06:01:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T06:01:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: Adding iSCSI gateway http://:@172.21.15.139:5000 to Dashboard 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi139"}]: dispatch 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi139"}]: dispatch 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2728367858' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/3587488312"}]: dispatch 2024-02-10T06:01:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[24727]: pgmap v86: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 204 B/s wr, 8 op/s 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/968128567"}]': finished 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: Adding iSCSI gateway http://:@172.21.15.139:5000 to Dashboard 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi139"}]: dispatch 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi139"}]: dispatch 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:54.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:54.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:54.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2728367858' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/3587488312"}]: dispatch 2024-02-10T06:01:54.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:53 smithi139 ceph-mon[29453]: pgmap v86: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 204 B/s wr, 8 op/s 2024-02-10T06:01:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/968128567"}]': finished 2024-02-10T06:01:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T06:01:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: Adding iSCSI gateway http://:@172.21.15.139:5000 to Dashboard 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi139"}]: dispatch 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi139"}]: dispatch 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2728367858' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/3587488312"}]: dispatch 2024-02-10T06:01:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:53 smithi150 ceph-mon[25582]: pgmap v86: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 204 B/s wr, 8 op/s 2024-02-10T06:01:55.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:55 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2728367858' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/3587488312"}]': finished 2024-02-10T06:01:55.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:55 smithi139 ceph-mon[24727]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T06:01:55.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:55 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2398691229' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/347598603"}]: dispatch 2024-02-10T06:01:55.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:55 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2728367858' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/3587488312"}]': finished 2024-02-10T06:01:55.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:55 smithi139 ceph-mon[29453]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T06:01:55.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:55 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2398691229' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/347598603"}]: dispatch 2024-02-10T06:01:55.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:55 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2728367858' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/3587488312"}]': finished 2024-02-10T06:01:55.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:55 smithi150 ceph-mon[25582]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T06:01:55.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:55 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2398691229' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/347598603"}]: dispatch 2024-02-10T06:01:56.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:56 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2398691229' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/347598603"}]': finished 2024-02-10T06:01:56.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:56 smithi139 ceph-mon[29453]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T06:01:56.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:56 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1089480845' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1349154726"}]: dispatch 2024-02-10T06:01:56.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:56 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1349154726"}]: dispatch 2024-02-10T06:01:56.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:56 smithi139 ceph-mon[29453]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T06:01:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:56 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2398691229' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/347598603"}]': finished 2024-02-10T06:01:56.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:56 smithi139 ceph-mon[24727]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T06:01:56.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:56 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1089480845' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1349154726"}]: dispatch 2024-02-10T06:01:56.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:56 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1349154726"}]: dispatch 2024-02-10T06:01:56.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:56 smithi139 ceph-mon[24727]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T06:01:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:56 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2398691229' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/347598603"}]': finished 2024-02-10T06:01:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:56 smithi150 ceph-mon[25582]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T06:01:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:56 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1089480845' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1349154726"}]: dispatch 2024-02-10T06:01:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:56 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1349154726"}]: dispatch 2024-02-10T06:01:56.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:56 smithi150 ceph-mon[25582]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T06:01:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:57 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1349154726"}]': finished 2024-02-10T06:01:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:57 smithi139 ceph-mon[24727]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T06:01:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:57 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2295517228' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2316715134"}]: dispatch 2024-02-10T06:01:57.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:57 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1349154726"}]': finished 2024-02-10T06:01:57.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:57 smithi139 ceph-mon[29453]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T06:01:57.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:57 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2295517228' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2316715134"}]: dispatch 2024-02-10T06:01:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:57 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1349154726"}]': finished 2024-02-10T06:01:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:57 smithi150 ceph-mon[25582]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T06:01:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:57 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2295517228' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2316715134"}]: dispatch 2024-02-10T06:01:58.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:01:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:01:58.487 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:58 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2295517228' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2316715134"}]': finished 2024-02-10T06:01:58.487 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:58 smithi139 ceph-mon[24727]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T06:01:58.488 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:58 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/381764007' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4005972016"}]: dispatch 2024-02-10T06:01:58.488 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:58 smithi139 ceph-mon[24727]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T06:01:58.488 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:58 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2295517228' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2316715134"}]': finished 2024-02-10T06:01:58.488 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:58 smithi139 ceph-mon[29453]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T06:01:58.488 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:58 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/381764007' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4005972016"}]: dispatch 2024-02-10T06:01:58.488 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:58 smithi139 ceph-mon[29453]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T06:01:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:58 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2295517228' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2316715134"}]': finished 2024-02-10T06:01:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:58 smithi150 ceph-mon[25582]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T06:01:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:58 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/381764007' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4005972016"}]: dispatch 2024-02-10T06:01:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:58 smithi150 ceph-mon[25582]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T06:01:59.500 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:59 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/381764007' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4005972016"}]': finished 2024-02-10T06:01:59.501 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:59 smithi139 ceph-mon[24727]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T06:01:59.501 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:59 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/336860676' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1349154726"}]: dispatch 2024-02-10T06:01:59.501 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:01:59 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1349154726"}]: dispatch 2024-02-10T06:01:59.501 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:59 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/381764007' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4005972016"}]': finished 2024-02-10T06:01:59.501 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:59 smithi139 ceph-mon[29453]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T06:01:59.501 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:59 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/336860676' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1349154726"}]: dispatch 2024-02-10T06:01:59.501 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:01:59 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1349154726"}]: dispatch 2024-02-10T06:01:59.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:59 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/381764007' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4005972016"}]': finished 2024-02-10T06:01:59.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:59 smithi150 ceph-mon[25582]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T06:01:59.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:59 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/336860676' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1349154726"}]: dispatch 2024-02-10T06:01:59.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:01:59 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1349154726"}]: dispatch 2024-02-10T06:02:00.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1349154726"}]': finished 2024-02-10T06:02:00.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[29453]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T06:02:00.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1214086040' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4231034715"}]: dispatch 2024-02-10T06:02:00.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4231034715"}]: dispatch 2024-02-10T06:02:00.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:00.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[29453]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:00.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1349154726"}]': finished 2024-02-10T06:02:00.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[24727]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T06:02:00.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1214086040' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4231034715"}]: dispatch 2024-02-10T06:02:00.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4231034715"}]: dispatch 2024-02-10T06:02:00.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:00.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-mon[24727]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:00.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:00 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1349154726"}]': finished 2024-02-10T06:02:00.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:00 smithi150 ceph-mon[25582]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T06:02:00.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:00 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1214086040' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4231034715"}]: dispatch 2024-02-10T06:02:00.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:00 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4231034715"}]: dispatch 2024-02-10T06:02:00.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:00 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:00.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:00 smithi150 ceph-mon[25582]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:01.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:02:00 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:00] "GET /metrics HTTP/1.1" 200 208042 "" "Prometheus/2.33.4" 2024-02-10T06:02:01.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:01 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4231034715"}]': finished 2024-02-10T06:02:01.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:01 smithi139 ceph-mon[24727]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T06:02:01.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:01 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2855423319' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1743661343"}]: dispatch 2024-02-10T06:02:01.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:01 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1743661343"}]: dispatch 2024-02-10T06:02:01.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:01 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4231034715"}]': finished 2024-02-10T06:02:01.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:01 smithi139 ceph-mon[29453]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T06:02:01.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:01 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2855423319' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1743661343"}]: dispatch 2024-02-10T06:02:01.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:01 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1743661343"}]: dispatch 2024-02-10T06:02:01.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:01 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/4231034715"}]': finished 2024-02-10T06:02:01.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:01 smithi150 ceph-mon[25582]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T06:02:01.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:01 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2855423319' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1743661343"}]: dispatch 2024-02-10T06:02:01.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:01 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1743661343"}]: dispatch 2024-02-10T06:02:02.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:02 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1743661343"}]': finished 2024-02-10T06:02:02.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:02 smithi139 ceph-mon[24727]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T06:02:02.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:02 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3969802119' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3393873192"}]: dispatch 2024-02-10T06:02:02.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:02 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3393873192"}]: dispatch 2024-02-10T06:02:02.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:02 smithi139 ceph-mon[24727]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:02.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:02 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1743661343"}]': finished 2024-02-10T06:02:02.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:02 smithi139 ceph-mon[29453]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T06:02:02.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:02 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3969802119' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3393873192"}]: dispatch 2024-02-10T06:02:02.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:02 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3393873192"}]: dispatch 2024-02-10T06:02:02.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:02 smithi139 ceph-mon[29453]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:02.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:02 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1743661343"}]': finished 2024-02-10T06:02:02.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:02 smithi150 ceph-mon[25582]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T06:02:02.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:02 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3969802119' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3393873192"}]: dispatch 2024-02-10T06:02:02.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:02 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3393873192"}]: dispatch 2024-02-10T06:02:02.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:02 smithi150 ceph-mon[25582]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:03.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:03 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3393873192"}]': finished 2024-02-10T06:02:03.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:03 smithi139 ceph-mon[24727]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T06:02:03.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:03 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2499539634' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3972026441"}]: dispatch 2024-02-10T06:02:03.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:03 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3972026441"}]: dispatch 2024-02-10T06:02:03.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:03 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3393873192"}]': finished 2024-02-10T06:02:03.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:03 smithi139 ceph-mon[29453]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T06:02:03.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:03 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2499539634' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3972026441"}]: dispatch 2024-02-10T06:02:03.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:03 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3972026441"}]: dispatch 2024-02-10T06:02:03.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:03 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3393873192"}]': finished 2024-02-10T06:02:03.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:03 smithi150 ceph-mon[25582]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T06:02:03.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:03 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2499539634' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3972026441"}]: dispatch 2024-02-10T06:02:03.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:03 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3972026441"}]: dispatch 2024-02-10T06:02:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:04 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3972026441"}]': finished 2024-02-10T06:02:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:04 smithi139 ceph-mon[24727]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T06:02:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:04 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/824226676' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1332696927"}]: dispatch 2024-02-10T06:02:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:04 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1332696927"}]: dispatch 2024-02-10T06:02:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:04 smithi139 ceph-mon[24727]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:04.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:04 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3972026441"}]': finished 2024-02-10T06:02:04.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:04 smithi139 ceph-mon[29453]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T06:02:04.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:04 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/824226676' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1332696927"}]: dispatch 2024-02-10T06:02:04.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:04 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1332696927"}]: dispatch 2024-02-10T06:02:04.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:04 smithi139 ceph-mon[29453]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:04 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3972026441"}]': finished 2024-02-10T06:02:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:04 smithi150 ceph-mon[25582]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T06:02:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:04 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/824226676' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1332696927"}]: dispatch 2024-02-10T06:02:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:04 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1332696927"}]: dispatch 2024-02-10T06:02:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:04 smithi150 ceph-mon[25582]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:05.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:05 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1332696927"}]': finished 2024-02-10T06:02:05.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:05 smithi139 ceph-mon[24727]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T06:02:05.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:05 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2522540125' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1743661343"}]: dispatch 2024-02-10T06:02:05.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:05 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1743661343"}]: dispatch 2024-02-10T06:02:05.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:05 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1332696927"}]': finished 2024-02-10T06:02:05.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:05 smithi139 ceph-mon[29453]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T06:02:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:05 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2522540125' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1743661343"}]: dispatch 2024-02-10T06:02:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:05 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1743661343"}]: dispatch 2024-02-10T06:02:05.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:05 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1332696927"}]': finished 2024-02-10T06:02:05.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:05 smithi150 ceph-mon[25582]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T06:02:05.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:05 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2522540125' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1743661343"}]: dispatch 2024-02-10T06:02:05.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:05 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1743661343"}]: dispatch 2024-02-10T06:02:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:06 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1743661343"}]': finished 2024-02-10T06:02:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:06 smithi139 ceph-mon[24727]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T06:02:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:06 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2820320546' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/687083216"}]: dispatch 2024-02-10T06:02:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:06 smithi139 ceph-mon[24727]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:06.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:06 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1743661343"}]': finished 2024-02-10T06:02:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:06 smithi139 ceph-mon[29453]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T06:02:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:06 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2820320546' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/687083216"}]: dispatch 2024-02-10T06:02:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:06 smithi139 ceph-mon[29453]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:06 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1743661343"}]': finished 2024-02-10T06:02:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:06 smithi150 ceph-mon[25582]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T06:02:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:06 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2820320546' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/687083216"}]: dispatch 2024-02-10T06:02:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:06 smithi150 ceph-mon[25582]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:07 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2820320546' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/687083216"}]': finished 2024-02-10T06:02:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:07 smithi139 ceph-mon[24727]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T06:02:07.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:07 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2820320546' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/687083216"}]': finished 2024-02-10T06:02:07.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:07 smithi139 ceph-mon[29453]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T06:02:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:07 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2820320546' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/687083216"}]': finished 2024-02-10T06:02:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:07 smithi150 ceph-mon[25582]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T06:02:08.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:02:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:02:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:08 smithi139 ceph-mon[24727]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:02:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:08 smithi139 ceph-mon[29453]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:02:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:08 smithi150 ceph-mon[25582]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:02:10.893 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:02:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:10] "GET /metrics HTTP/1.1" 200 208107 "" "Prometheus/2.33.4" 2024-02-10T06:02:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:10 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:10 smithi139 ceph-mon[29453]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:10 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:10 smithi139 ceph-mon[24727]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:10 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:10 smithi150 ceph-mon[25582]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:12 smithi139 ceph-mon[24727]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:02:13.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:12 smithi139 ceph-mon[29453]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:02:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:12 smithi150 ceph-mon[25582]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:02:15.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:14 smithi139 ceph-mon[24727]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:15.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:14 smithi139 ceph-mon[29453]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:15.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:14 smithi150 ceph-mon[25582]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:17.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:16 smithi139 ceph-mon[24727]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:02:17.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:16 smithi139 ceph-mon[29453]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:02:17.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:16 smithi150 ceph-mon[25582]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:02:18.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:02:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:02:19.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:18 smithi139 ceph-mon[24727]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-02-10T06:02:19.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:18 smithi139 ceph-mon[29453]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-02-10T06:02:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:18 smithi150 ceph-mon[25582]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-02-10T06:02:20.903 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:02:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:20] "GET /metrics HTTP/1.1" 200 208078 "" "Prometheus/2.33.4" 2024-02-10T06:02:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:20 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:20 smithi139 ceph-mon[24727]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:20 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:20 smithi139 ceph-mon[29453]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:20 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:20 smithi150 ceph-mon[25582]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:23.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:22 smithi139 ceph-mon[24727]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:23.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:22 smithi139 ceph-mon[29453]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:22 smithi150 ceph-mon[25582]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:25.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:24 smithi139 ceph-mon[24727]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:25.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:24 smithi139 ceph-mon[29453]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:25.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:24 smithi150 ceph-mon[25582]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:27.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:26 smithi139 ceph-mon[24727]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:27.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:26 smithi139 ceph-mon[29453]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:27.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:26 smithi150 ceph-mon[25582]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:28.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:02:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:02:29.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:28 smithi139 ceph-mon[24727]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:29.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:28 smithi139 ceph-mon[29453]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:29.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:28 smithi150 ceph-mon[25582]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:30.906 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:02:30 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:30] "GET /metrics HTTP/1.1" 200 208078 "" "Prometheus/2.33.4" 2024-02-10T06:02:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:30 smithi139 ceph-mon[29453]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:30 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:30 smithi139 ceph-mon[24727]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:30 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:30 smithi150 ceph-mon[25582]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:30 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:33.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:32 smithi139 ceph-mon[24727]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:32 smithi139 ceph-mon[29453]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:32 smithi150 ceph-mon[25582]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:35.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:34 smithi139 ceph-mon[24727]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:35.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:34 smithi139 ceph-mon[29453]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:35.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:34 smithi150 ceph-mon[25582]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:36 smithi139 ceph-mon[24727]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:37.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:36 smithi139 ceph-mon[29453]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:37.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:36 smithi150 ceph-mon[25582]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:38.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:02:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:02:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:38 smithi139 ceph-mon[24727]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:39.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:38 smithi139 ceph-mon[29453]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:39.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:38 smithi150 ceph-mon[25582]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:40.907 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:02:40 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:40] "GET /metrics HTTP/1.1" 200 208052 "" "Prometheus/2.33.4" 2024-02-10T06:02:41.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:40 smithi139 ceph-mon[24727]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:41.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:40 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:41.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:40 smithi139 ceph-mon[29453]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:41.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:40 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:40 smithi150 ceph-mon[25582]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:40 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:43.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:42 smithi139 ceph-mon[24727]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:43.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:42 smithi139 ceph-mon[29453]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:43.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:42 smithi150 ceph-mon[25582]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:45.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:44 smithi150 ceph-mon[25582]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:45.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:44 smithi139 ceph-mon[24727]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:45.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:44 smithi139 ceph-mon[29453]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:47.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:46 smithi139 ceph-mon[24727]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:47.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:46 smithi139 ceph-mon[29453]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:47.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:46 smithi150 ceph-mon[25582]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:48.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:02:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:02:49.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:48 smithi139 ceph-mon[24727]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:49.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:48 smithi139 ceph-mon[29453]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:49.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:48 smithi150 ceph-mon[25582]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:02:50.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:02:50.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:02:50.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:02:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:02:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:02:50.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:02:50.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:02:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:02:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:02:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:02:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:02:50.916 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:02:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:50] "GET /metrics HTTP/1.1" 200 208030 "" "Prometheus/2.33.4" 2024-02-10T06:02:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:50 smithi139 ceph-mon[24727]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:50 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:51.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:50 smithi139 ceph-mon[29453]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:51.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:50 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:50 smithi150 ceph-mon[25582]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:50 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:53.212 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:52 smithi139 ceph-mon[24727]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:53.213 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:52 smithi139 ceph-mon[29453]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:53.213 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:52 smithi150 ceph-mon[25582]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:02:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:02:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:02:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:53 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:02:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:02:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:02:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:02:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:53 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:02:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:02:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:02:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:02:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:53 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:02:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:54 smithi139 ceph-mon[24727]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:55.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:54 smithi139 ceph-mon[29453]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:55.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:54 smithi150 ceph-mon[25582]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:56 smithi139 ceph-mon[24727]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:57.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:56 smithi139 ceph-mon[29453]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:56 smithi150 ceph-mon[25582]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:58.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:02:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:02:59.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:02:58 smithi139 ceph-mon[24727]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:59.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:02:58 smithi139 ceph-mon[29453]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:02:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:02:58 smithi150 ceph-mon[25582]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:00.936 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:03:00 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:00] "GET /metrics HTTP/1.1" 200 208030 "" "Prometheus/2.33.4" 2024-02-10T06:03:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:00 smithi139 ceph-mon[24727]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:00 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:01.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:00 smithi139 ceph-mon[29453]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:01.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:00 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:00 smithi150 ceph-mon[25582]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:00 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:02 smithi139 ceph-mon[24727]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:03.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:02 smithi139 ceph-mon[29453]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:02 smithi150 ceph-mon[25582]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:05.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:04 smithi150 ceph-mon[25582]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:05.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:04 smithi139 ceph-mon[24727]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:05.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:04 smithi139 ceph-mon[29453]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:07.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:06 smithi139 ceph-mon[24727]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:07.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:06 smithi139 ceph-mon[29453]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:07.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:06 smithi150 ceph-mon[25582]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:08.125 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:03:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:03:09.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:08 smithi139 ceph-mon[29453]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:09.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:08 smithi139 ceph-mon[24727]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:08 smithi150 ceph-mon[25582]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:10.917 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:03:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:10] "GET /metrics HTTP/1.1" 200 208033 "" "Prometheus/2.33.4" 2024-02-10T06:03:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:10 smithi139 ceph-mon[24727]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:10 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:10 smithi139 ceph-mon[29453]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:10 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:10 smithi150 ceph-mon[25582]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:10 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:12 smithi139 ceph-mon[24727]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:13.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:12 smithi139 ceph-mon[29453]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:12 smithi150 ceph-mon[25582]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:15.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:14 smithi150 ceph-mon[25582]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:15.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:14 smithi139 ceph-mon[24727]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:15.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:14 smithi139 ceph-mon[29453]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:17.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:16 smithi139 ceph-mon[24727]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:17.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:16 smithi139 ceph-mon[29453]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:17.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:16 smithi150 ceph-mon[25582]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:18.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:03:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:03:19.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:18 smithi139 ceph-mon[24727]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:19.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:18 smithi139 ceph-mon[29453]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:18 smithi150 ceph-mon[25582]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:20.923 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:03:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:20] "GET /metrics HTTP/1.1" 200 208031 "" "Prometheus/2.33.4" 2024-02-10T06:03:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:20 smithi139 ceph-mon[24727]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:20 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:20 smithi139 ceph-mon[29453]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:20 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:20 smithi150 ceph-mon[25582]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:20 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:23.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:22 smithi139 ceph-mon[24727]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:23.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:22 smithi139 ceph-mon[29453]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:22 smithi150 ceph-mon[25582]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:25.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:24 smithi150 ceph-mon[25582]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:25.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:24 smithi139 ceph-mon[24727]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:25.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:24 smithi139 ceph-mon[29453]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:27.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:26 smithi139 ceph-mon[24727]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:27.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:26 smithi139 ceph-mon[29453]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:27.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:26 smithi150 ceph-mon[25582]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:28.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:03:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:03:29.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:28 smithi139 ceph-mon[24727]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:29.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:28 smithi139 ceph-mon[29453]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:28 smithi150 ceph-mon[25582]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:30.927 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:03:30 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:30] "GET /metrics HTTP/1.1" 200 208031 "" "Prometheus/2.33.4" 2024-02-10T06:03:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:30 smithi139 ceph-mon[24727]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:30 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:30 smithi139 ceph-mon[29453]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:30 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:30 smithi150 ceph-mon[25582]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:30 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:33.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:32 smithi139 ceph-mon[24727]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:32 smithi139 ceph-mon[29453]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:32 smithi150 ceph-mon[25582]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:35.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:34 smithi150 ceph-mon[25582]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:35.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:34 smithi139 ceph-mon[24727]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:35.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:34 smithi139 ceph-mon[29453]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:36 smithi139 ceph-mon[24727]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:37.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:36 smithi139 ceph-mon[29453]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:37.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:36 smithi150 ceph-mon[25582]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:38.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:03:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:03:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:38 smithi139 ceph-mon[24727]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:39.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:38 smithi139 ceph-mon[29453]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:39.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:38 smithi150 ceph-mon[25582]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:40.927 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:03:40 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:40] "GET /metrics HTTP/1.1" 200 208038 "" "Prometheus/2.33.4" 2024-02-10T06:03:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:40 smithi139 ceph-mon[24727]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:40 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:40 smithi139 ceph-mon[29453]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:40 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:40 smithi150 ceph-mon[25582]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:40 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:43.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:42 smithi139 ceph-mon[24727]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:43.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:42 smithi139 ceph-mon[29453]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:43.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:42 smithi150 ceph-mon[25582]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:45.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:44 smithi150 ceph-mon[25582]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:45.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:44 smithi139 ceph-mon[24727]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:45.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:44 smithi139 ceph-mon[29453]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:47.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:46 smithi139 ceph-mon[24727]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:47.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:46 smithi139 ceph-mon[29453]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:47.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:46 smithi150 ceph-mon[25582]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:48.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:03:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:03:49.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:48 smithi139 ceph-mon[24727]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:49.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:48 smithi139 ceph-mon[29453]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:49.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:48 smithi150 ceph-mon[25582]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:03:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:03:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:03:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:03:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:03:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:03:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:03:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:03:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:03:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:03:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:03:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:03:50.942 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:03:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:50] "GET /metrics HTTP/1.1" 200 208019 "" "Prometheus/2.33.4" 2024-02-10T06:03:51.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:50 smithi139 ceph-mon[29453]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:51.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:50 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:50 smithi139 ceph-mon[24727]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:51.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:50 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:50 smithi150 ceph-mon[25582]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:50 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:53.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:52 smithi139 ceph-mon[24727]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:53.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:52 smithi139 ceph-mon[29453]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:52 smithi150 ceph-mon[25582]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:03:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:03:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:53 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:03:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:53 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:03:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:03:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:03:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:53 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:03:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:53 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:03:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:03:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:03:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:53 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:03:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:53 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:03:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:54 smithi139 ceph-mon[24727]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:55.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:54 smithi139 ceph-mon[29453]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:55.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:54 smithi150 ceph-mon[25582]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:56 smithi139 ceph-mon[24727]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:57.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:56 smithi139 ceph-mon[29453]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:56 smithi150 ceph-mon[25582]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:58.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:03:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:03:59.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:03:58 smithi139 ceph-mon[24727]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:59.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:03:58 smithi139 ceph-mon[29453]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:03:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:03:58 smithi150 ceph-mon[25582]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:00.958 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:04:00 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:00] "GET /metrics HTTP/1.1" 200 208019 "" "Prometheus/2.33.4" 2024-02-10T06:04:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:00 smithi139 ceph-mon[24727]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:00 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:01.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:00 smithi139 ceph-mon[29453]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:01.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:00 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:00 smithi150 ceph-mon[25582]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:00 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:02 smithi139 ceph-mon[24727]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:03.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:02 smithi139 ceph-mon[29453]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:02 smithi150 ceph-mon[25582]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:05.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:04 smithi150 ceph-mon[25582]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:05.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:04 smithi139 ceph-mon[24727]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:05.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:04 smithi139 ceph-mon[29453]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:07.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:06 smithi139 ceph-mon[24727]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:07.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:06 smithi139 ceph-mon[29453]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:07.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:06 smithi150 ceph-mon[25582]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:08.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:04:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:04:09.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:08 smithi139 ceph-mon[24727]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:09.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:08 smithi139 ceph-mon[29453]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:09.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:08 smithi150 ceph-mon[25582]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:10.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:04:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:10] "GET /metrics HTTP/1.1" 200 208020 "" "Prometheus/2.33.4" 2024-02-10T06:04:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:10 smithi139 ceph-mon[24727]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:10 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:10 smithi139 ceph-mon[29453]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:10 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:10 smithi150 ceph-mon[25582]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:10 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:12 smithi139 ceph-mon[24727]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:13.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:12 smithi139 ceph-mon[29453]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:12 smithi150 ceph-mon[25582]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:15.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:14 smithi150 ceph-mon[25582]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:15.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:14 smithi139 ceph-mon[24727]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:15.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:14 smithi139 ceph-mon[29453]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:17.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:16 smithi139 ceph-mon[24727]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:17.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:16 smithi139 ceph-mon[29453]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:17.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:16 smithi150 ceph-mon[25582]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:18.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:04:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:04:19.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:18 smithi139 ceph-mon[24727]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:19.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:18 smithi139 ceph-mon[29453]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:18 smithi150 ceph-mon[25582]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:20.940 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:04:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:20] "GET /metrics HTTP/1.1" 200 207965 "" "Prometheus/2.33.4" 2024-02-10T06:04:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:20 smithi139 ceph-mon[24727]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:20 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:20 smithi139 ceph-mon[29453]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:20 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:20 smithi150 ceph-mon[25582]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:20 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:23.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:22 smithi139 ceph-mon[24727]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:23.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:22 smithi139 ceph-mon[29453]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:22 smithi150 ceph-mon[25582]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:25.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:24 smithi150 ceph-mon[25582]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:25.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:24 smithi139 ceph-mon[24727]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:25.383 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:24 smithi139 ceph-mon[29453]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:27.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:26 smithi139 ceph-mon[24727]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:27.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:26 smithi139 ceph-mon[29453]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:27.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:26 smithi150 ceph-mon[25582]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:28.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:04:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:04:29.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:28 smithi139 ceph-mon[29453]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:29.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:28 smithi139 ceph-mon[24727]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:28 smithi150 ceph-mon[25582]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:30.943 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:04:30 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:30] "GET /metrics HTTP/1.1" 200 207965 "" "Prometheus/2.33.4" 2024-02-10T06:04:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:30 smithi139 ceph-mon[29453]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:30 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:30 smithi139 ceph-mon[24727]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:30 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:30 smithi150 ceph-mon[25582]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:30 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:33.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:32 smithi139 ceph-mon[24727]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:32 smithi139 ceph-mon[29453]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:32 smithi150 ceph-mon[25582]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:35.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:34 smithi150 ceph-mon[25582]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:35.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:34 smithi139 ceph-mon[24727]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:35.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:34 smithi139 ceph-mon[29453]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:36 smithi139 ceph-mon[24727]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:37.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:36 smithi139 ceph-mon[29453]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:37.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:36 smithi150 ceph-mon[25582]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:38.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:04:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:04:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:38 smithi139 ceph-mon[24727]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:39.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:38 smithi139 ceph-mon[29453]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:39.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:38 smithi150 ceph-mon[25582]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:40.945 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:04:40 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:40] "GET /metrics HTTP/1.1" 200 207976 "" "Prometheus/2.33.4" 2024-02-10T06:04:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:40 smithi139 ceph-mon[29453]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:40 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:40 smithi139 ceph-mon[24727]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:40 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:40 smithi150 ceph-mon[25582]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:40 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:43.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:42 smithi139 ceph-mon[24727]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:43.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:42 smithi139 ceph-mon[29453]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:43.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:42 smithi150 ceph-mon[25582]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:45.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:44 smithi150 ceph-mon[25582]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:45.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:44 smithi139 ceph-mon[24727]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:45.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:44 smithi139 ceph-mon[29453]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:47.247 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:46 smithi139 ceph-mon[24727]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:47.247 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:46 smithi139 ceph-mon[29453]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:47.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:46 smithi150 ceph-mon[25582]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:47.642 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-10T06:04:48.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:04:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:04:49.215 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:48 smithi139 ceph-mon[24727]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:49.216 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:48 smithi139 ceph-mon[29453]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:49.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:48 smithi150 ceph-mon[25582]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:04:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:04:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:04:50.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:04:50.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:04:50.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:04:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:04:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:04:50.134 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-10T06:04:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:04:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:04:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:04:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:04:50.663 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:04:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:50] "GET /metrics HTTP/1.1" 200 207970 "" "Prometheus/2.33.4" 2024-02-10T06:04:51.316 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:50 smithi139 ceph-mon[24727]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:51.316 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:50 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:51.316 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:50 smithi139 ceph-mon[29453]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:51.316 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:50 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:50 smithi150 ceph-mon[25582]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:50 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:52.750 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-10T06:04:53.059 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:52 smithi139 ceph-mon[24727]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:53.059 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:52 smithi139 ceph-mon[29453]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:52 smithi150 ceph-mon[25582]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:55.029 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:54 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:04:55.029 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:54 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:04:55.029 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:54 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:04:55.029 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:54 smithi139 ceph-mon[24727]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:55.029 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:54 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:04:55.030 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:54 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:04:55.030 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:54 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:04:55.030 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:54 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:04:55.030 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:54 smithi139 ceph-mon[29453]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:55.030 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:54 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:04:55.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:54 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:04:55.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:54 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:04:55.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:54 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:04:55.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:54 smithi150 ceph-mon[25582]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:55.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:54 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:04:55.344 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:04:57.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:56 smithi139 ceph-mon[24727]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:57.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:56 smithi139 ceph-mon[29453]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:57.132 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (3m) 3m ago 4m 17.6M - ba2b418f427c adb2e2c020a1 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (3m) 3m ago 3m 45.6M - 8.3.5 dad864ee21e9 db9dee5d8d83 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (3m) 3m ago 3m 73.7M - 3.5 e1d6a67b021e cc86a3c37228 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443 running (8m) 3m ago 8m 413M - 17.2.0 e1d6a67b021e 62310c7d2542 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:9283 running (10m) 3m ago 10m 454M - 17.2.0 e1d6a67b021e 5d5f244aa01f 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (10m) 3m ago 10m 48.8M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (8m) 3m ago 8m 38.8M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (8m) 3m ago 8m 39.5M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (4m) 3m ago 4m 16.3M - 1dbe0e931976 5b38421d9397 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (4m) 3m ago 4m 14.5M - 1dbe0e931976 c79c2778bd87 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (8m) 3m ago 8m 46.6M 4016M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (7m) 3m ago 7m 48.0M 4016M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (7m) 3m ago 7m 46.7M 4016M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (6m) 3m ago 6m 45.4M 4016M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (6m) 3m ago 6m 44.7M 4272M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:04:57.133 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (6m) 3m ago 6m 45.9M 4272M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:04:57.134 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (5m) 3m ago 5m 44.1M 4272M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:04:57.134 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (5m) 3m ago 5m 47.2M 4272M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:04:57.134 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (3m) 3m ago 4m 42.9M - 514e6a882f6e 81a684e8f18f 2024-02-10T06:04:57.134 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (3m) 3m ago 3m 68.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:04:57.134 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (3m) 3m ago 3m 67.4M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:04:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:56 smithi150 ceph-mon[25582]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:57.822 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T06:04:58.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:04:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:04:59.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:58 smithi139 ceph-mon[24727]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:04:59.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:58 smithi139 ceph-mon[24727]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:59.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:58 smithi139 ceph-mon[29453]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:04:59.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:58 smithi139 ceph-mon[29453]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:58 smithi150 ceph-mon[25582]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:04:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:58 smithi150 ceph-mon[25582]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:04:59.903 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:04:59.903 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:04:59.903 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T06:04:59.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:04:59.903 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:04:59.903 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:04:59.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:04:59.903 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:04:59.904 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:04:59.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:04:59.904 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": {}, 2024-02-10T06:04:59.904 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:04:59.904 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:04:59.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:04:59.904 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:04:59.904 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-02-10T06:04:59.904 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:04:59.904 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:05:00.213 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:04:59 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2103312666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:05:00.213 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:04:59 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2103312666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:05:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:04:59 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2103312666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:05:00.596 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph -s' 2024-02-10T06:05:00.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:05:00 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:00] "GET /metrics HTTP/1.1" 200 207970 "" "Prometheus/2.33.4" 2024-02-10T06:05:01.167 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:00 smithi139 ceph-mon[24727]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:01.167 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:00 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:01.167 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:00 smithi139 ceph-mon[29453]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:01.167 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:00 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:00 smithi150 ceph-mon[25582]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:00 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:02.672 INFO:teuthology.orchestra.run.smithi139.stdout: cluster: 2024-02-10T06:05:02.672 INFO:teuthology.orchestra.run.smithi139.stdout: id: ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:05:02.672 INFO:teuthology.orchestra.run.smithi139.stdout: health: HEALTH_OK 2024-02-10T06:05:02.672 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: services: 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: mon: 3 daemons, quorum a,c,b (age 8m) 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: mgr: y(active, since 5m), standbys: x 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: data: 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: pools: 6 pools, 161 pgs 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: objects: 209 objects, 585 KiB 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: usage: 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: pgs: 161 active+clean 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: io: 2024-02-10T06:05:02.673 INFO:teuthology.orchestra.run.smithi139.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-02-10T06:05:02.674 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:05:03.293 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:02 smithi139 ceph-mon[24727]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:03.294 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:02 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/4266489989' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:05:03.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:02 smithi139 ceph-mon[29453]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:03.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:02 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/4266489989' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:05:03.318 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ls' 2024-02-10T06:05:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:02 smithi150 ceph-mon[25582]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:02 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/4266489989' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:05:05.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:04 smithi139 ceph-mon[24727]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:05.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:04 smithi139 ceph-mon[29453]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:05.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:04 smithi150 ceph-mon[25582]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:05.250 INFO:teuthology.orchestra.run.smithi139.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-10T06:05:05.250 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager ?:9093,9094 1/1 3m ago 5m smithi139=a;count:1 2024-02-10T06:05:05.251 INFO:teuthology.orchestra.run.smithi139.stdout:grafana ?:3000 1/1 3m ago 5m smithi150=a;count:1 2024-02-10T06:05:05.251 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo 1/1 3m ago 3m count:1 2024-02-10T06:05:05.251 INFO:teuthology.orchestra.run.smithi139.stdout:mgr 2/2 3m ago 8m smithi139=y;smithi150=x;count:2 2024-02-10T06:05:05.251 INFO:teuthology.orchestra.run.smithi139.stdout:mon 3/3 3m ago 9m smithi139:172.21.15.139=a;smithi139:[v2:172.21.15.139:3301,v1:172.21.15.139:6790]=c;smithi150:172.21.15.150=b;count:3 2024-02-10T06:05:05.251 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter ?:9100 2/2 3m ago 5m smithi139=a;smithi150=b;count:2 2024-02-10T06:05:05.251 INFO:teuthology.orchestra.run.smithi139.stdout:osd 8 3m ago - 2024-02-10T06:05:05.251 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus ?:9095 1/1 3m ago 5m smithi150=a;count:1 2024-02-10T06:05:05.251 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo ?:8000 2/2 3m ago 3m count:2 2024-02-10T06:05:05.865 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-10T06:05:07.222 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:06 smithi139 ceph-mon[24727]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:05:07.223 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:06 smithi139 ceph-mon[24727]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:07.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:06 smithi139 ceph-mon[29453]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:05:07.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:06 smithi139 ceph-mon[29453]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:06 smithi150 ceph-mon[25582]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:05:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:06 smithi150 ceph-mon[25582]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:08.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:07 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/182964423' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:05:08.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:05:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:05:08.185 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled to redeploy mgr.x on host 'smithi150' 2024-02-10T06:05:08.255 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:07 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/182964423' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:05:08.256 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:07 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/182964423' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:05:08.873 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps --refresh' 2024-02-10T06:05:09.202 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:08 smithi139 ceph-mon[24727]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:09.203 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:08 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:09.203 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:08 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:05:09.203 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:08 smithi139 ceph-mon[29453]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:09.203 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:08 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:09.203 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:08 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:05:09.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:08 smithi150 ceph-mon[25582]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:09.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:08 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:09.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:08 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:05:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:09 smithi139 ceph-mon[24727]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:05:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:09 smithi139 ceph-mon[24727]: Schedule redeploy daemon mgr.x 2024-02-10T06:05:10.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:09 smithi139 ceph-mon[29453]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:05:10.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:09 smithi139 ceph-mon[29453]: Schedule redeploy daemon mgr.x 2024-02-10T06:05:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:09 smithi150 ceph-mon[25582]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:05:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:09 smithi150 ceph-mon[25582]: Schedule redeploy daemon mgr.x 2024-02-10T06:05:10.638 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:05:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:10] "GET /metrics HTTP/1.1" 200 207961 "" "Prometheus/2.33.4" 2024-02-10T06:05:11.174 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:05:11.174 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (3m) 3m ago 4m 17.6M - ba2b418f427c adb2e2c020a1 2024-02-10T06:05:11.174 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (4m) 3m ago 4m 45.6M - 8.3.5 dad864ee21e9 db9dee5d8d83 2024-02-10T06:05:11.174 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (3m) 3m ago 3m 73.7M - 3.5 e1d6a67b021e cc86a3c37228 2024-02-10T06:05:11.174 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443 running (8m) 3m ago 8m 413M - 17.2.0 e1d6a67b021e 62310c7d2542 2024-02-10T06:05:11.174 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:9283 running (10m) 3m ago 10m 454M - 17.2.0 e1d6a67b021e 5d5f244aa01f 2024-02-10T06:05:11.174 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (10m) 3m ago 10m 48.8M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:05:11.174 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (8m) 3m ago 8m 38.8M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:05:11.174 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (9m) 3m ago 9m 39.5M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:05:11.174 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (4m) 3m ago 4m 16.3M - 1dbe0e931976 5b38421d9397 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (4m) 3m ago 4m 14.5M - 1dbe0e931976 c79c2778bd87 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (8m) 3m ago 8m 46.6M 4016M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (8m) 3m ago 8m 48.0M 4016M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (7m) 3m ago 7m 46.7M 4016M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (7m) 3m ago 7m 45.4M 4016M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (6m) 3m ago 6m 44.7M 4272M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (6m) 3m ago 6m 45.9M 4272M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (6m) 3m ago 6m 44.1M 4272M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (5m) 3m ago 5m 47.2M 4272M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (3m) 3m ago 5m 42.9M - 514e6a882f6e 81a684e8f18f 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (3m) 3m ago 3m 68.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:05:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (3m) 3m ago 3m 67.4M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:05:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:10 smithi139 ceph-mon[24727]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:10 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:10 smithi139 ceph-mon[29453]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:10 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:10 smithi150 ceph-mon[25582]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:10 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:11.965 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'sleep 180' 2024-02-10T06:05:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:12 smithi139 ceph-mon[24727]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:05:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:12 smithi139 ceph-mon[24727]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:13.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:12 smithi139 ceph-mon[29453]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:05:13.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:12 smithi139 ceph-mon[29453]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:12 smithi150 ceph-mon[25582]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:05:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:12 smithi150 ceph-mon[25582]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:15.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:14 smithi150 ceph-mon[25582]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:15.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:14 smithi139 ceph-mon[24727]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:15.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:14 smithi139 ceph-mon[29453]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:17.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:16 smithi139 ceph-mon[24727]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:17.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:16 smithi139 ceph-mon[29453]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:17.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:16 smithi150 ceph-mon[25582]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:18.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:05:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:05:19.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:18 smithi139 ceph-mon[24727]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:19.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:18 smithi139 ceph-mon[29453]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:18 smithi150 ceph-mon[25582]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:20.955 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:05:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:20] "GET /metrics HTTP/1.1" 200 207971 "" "Prometheus/2.33.4" 2024-02-10T06:05:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:20 smithi139 ceph-mon[24727]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:21 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:20 smithi139 ceph-mon[29453]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:21 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:20 smithi150 ceph-mon[25582]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:20 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:23.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:22 smithi139 ceph-mon[24727]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:23.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:22 smithi139 ceph-mon[29453]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:22 smithi150 ceph-mon[25582]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:25.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:24 smithi139 ceph-mon[24727]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:25.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:24 smithi139 ceph-mon[29453]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:25.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:24 smithi150 ceph-mon[25582]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:27.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:26 smithi139 ceph-mon[24727]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:27.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:26 smithi139 ceph-mon[29453]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:27.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:26 smithi150 ceph-mon[25582]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:28.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:05:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:05:29.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:28 smithi139 ceph-mon[24727]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:29.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:28 smithi139 ceph-mon[29453]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:28 smithi150 ceph-mon[25582]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:30.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:05:30 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:30] "GET /metrics HTTP/1.1" 200 207971 "" "Prometheus/2.33.4" 2024-02-10T06:05:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:30 smithi139 ceph-mon[24727]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:30 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:30 smithi139 ceph-mon[29453]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:30 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:30 smithi150 ceph-mon[25582]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:30 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:33.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:32 smithi139 ceph-mon[24727]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:32 smithi139 ceph-mon[29453]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:32 smithi150 ceph-mon[25582]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:35.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:34 smithi139 ceph-mon[24727]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:35.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:34 smithi139 ceph-mon[29453]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:35.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:34 smithi150 ceph-mon[25582]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:36 smithi139 ceph-mon[24727]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:37.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:36 smithi139 ceph-mon[29453]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:37.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:36 smithi150 ceph-mon[25582]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:38.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:05:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:05:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:38 smithi139 ceph-mon[24727]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:39.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:38 smithi139 ceph-mon[29453]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:39.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:38 smithi150 ceph-mon[25582]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:40.613 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:40.613 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:05:40.613 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:05:40.613 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:40.613 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[24727]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:40.613 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:40.614 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:40.614 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:05:40.614 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:05:40.614 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:40.614 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[29453]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:40.614 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:40.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:40 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:40.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:40 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:05:40.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:40 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:05:40.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:40 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:40.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:40 smithi150 ceph-mon[25582]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:40.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:40 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:41.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:05:40 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:40] "GET /metrics HTTP/1.1" 200 207966 "" "Prometheus/2.33.4" 2024-02-10T06:05:43.229 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:42 smithi150 ceph-mon[25582]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:43.229 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:42 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:43.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:42 smithi139 ceph-mon[24727]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:43.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:42 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:43.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:42 smithi139 ceph-mon[29453]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:43.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:42 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:05:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:05:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:05:44.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:05:44.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[24727]: Deploying daemon mgr.x on smithi150 2024-02-10T06:05:44.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[24727]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:05:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:05:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:05:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:05:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[29453]: Deploying daemon mgr.x on smithi150 2024-02-10T06:05:44.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:44 smithi139 ceph-mon[29453]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:44 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:44 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:05:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:44 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:05:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:44 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:05:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:44 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:05:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:44 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:05:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:44 smithi150 ceph-mon[25582]: Deploying daemon mgr.x on smithi150 2024-02-10T06:05:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:44 smithi150 ceph-mon[25582]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:47.327 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:46 smithi150 ceph-mon[25582]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:47.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:46 smithi139 ceph-mon[24727]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:47.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:46 smithi139 ceph-mon[29453]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:47.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:05:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:05:49.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:48 smithi139 ceph-mon[24727]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:49.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:48 smithi139 ceph-mon[29453]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:49.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:48 smithi150 ceph-mon[25582]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:05:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:05:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:05:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:05:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:05:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:05:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:05:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:05:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:05:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:05:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:05:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:05:50.966 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:05:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:50] "GET /metrics HTTP/1.1" 200 207982 "" "Prometheus/2.33.4" 2024-02-10T06:05:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:50 smithi139 ceph-mon[24727]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:50 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:51.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:50 smithi139 ceph-mon[29453]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:51.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:50 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:50 smithi150 ceph-mon[25582]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:50 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:53.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:52 smithi139 ceph-mon[24727]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:53.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:52 smithi139 ceph-mon[29453]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:52 smithi150 ceph-mon[25582]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:53.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:05:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:05:53.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:05:53.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:05:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:05:53.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:05:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:54 smithi139 ceph-mon[24727]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:55.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:54 smithi139 ceph-mon[29453]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:55.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:54 smithi150 ceph-mon[25582]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:56 smithi139 ceph-mon[24727]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:57.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:56 smithi139 ceph-mon[29453]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:56 smithi150 ceph-mon[25582]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:05:58.017 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:05:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:05:59.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:05:58 smithi139 ceph-mon[24727]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:59.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:05:58 smithi139 ceph-mon[29453]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:05:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:05:58 smithi150 ceph-mon[25582]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:00.969 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:06:00 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:06:00] "GET /metrics HTTP/1.1" 200 207982 "" "Prometheus/2.33.4" 2024-02-10T06:06:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:00 smithi139 ceph-mon[24727]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:00 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:00 smithi139 ceph-mon[29453]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:00 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:00 smithi150 ceph-mon[25582]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:00 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:02 smithi139 ceph-mon[24727]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:03.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:02 smithi139 ceph-mon[29453]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:02 smithi150 ceph-mon[25582]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:06:03.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:06:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:03.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:06:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:03.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:06:05.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:04 smithi139 ceph-mon[24727]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:05.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:04 smithi139 ceph-mon[29453]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:05.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:04 smithi150 ceph-mon[25582]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:07.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:06 smithi139 ceph-mon[24727]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:07.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:06 smithi139 ceph-mon[29453]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:07.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:06 smithi150 ceph-mon[25582]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:08.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[26763]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:06:09.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:08 smithi139 ceph-mon[24727]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:09.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:08 smithi139 ceph-mon[29453]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:09.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:08 smithi150 ceph-mon[25582]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:10.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:06:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:06:10] "GET /metrics HTTP/1.1" 200 207981 "" "Prometheus/2.33.4" 2024-02-10T06:06:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:10 smithi139 ceph-mon[29453]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:10 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:10 smithi139 ceph-mon[24727]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:10 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:10 smithi150 ceph-mon[25582]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:10 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:13.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:12 smithi150 ceph-mon[25582]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:12 smithi139 ceph-mon[24727]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:13.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:12 smithi139 ceph-mon[29453]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:13.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:06:13.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:06:13.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:13.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:06:13.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:13.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:06:13.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:13 smithi150 systemd[1]: Stopping Ceph mgr.x for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:06:13.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:13 smithi150 bash[45639]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr.x" found: no such container 2024-02-10T06:06:14.214 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:13 smithi150 podman[45652]: 2024-02-10 06:06:13.941737026 +0000 UTC m=+0.122753371 container died 62310c7d2542ae5d8abfb6d454b68f29af33baeaaba327e93662f2ebdc1c722f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, ceph=True, version=8, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.openshift.tags=base centos centos-stream, vcs-type=git, RELEASE=HEAD) 2024-02-10T06:06:14.572 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:14 smithi150 podman[45652]: 2024-02-10 06:06:14.305059094 +0000 UTC m=+0.486075443 container cleanup 62310c7d2542ae5d8abfb6d454b68f29af33baeaaba327e93662f2ebdc1c722f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, version=8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.component=centos-stream-container, release=754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, vcs-type=git, io.buildah.version=1.19.8, RELEASE=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-02-10T06:06:14.573 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:14 smithi150 bash[45652]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x 2024-02-10T06:06:14.573 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:14 smithi150 bash[45680]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr.x" found: no such container 2024-02-10T06:06:14.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:14 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:14.200Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=2 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused" 2024-02-10T06:06:14.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:14 smithi150 podman[45666]: 2024-02-10 06:06:14.573106871 +0000 UTC m=+0.627399975 container remove 62310c7d2542ae5d8abfb6d454b68f29af33baeaaba327e93662f2ebdc1c722f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8) 2024-02-10T06:06:14.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:14 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:06:14.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:14 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-10T06:06:14.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:14 smithi150 systemd[1]: Stopped Ceph mgr.x for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:06:14.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:14 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.x.service: Consumed 17.074s CPU time. 2024-02-10T06:06:14.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:14 smithi150 systemd[1]: Starting Ceph mgr.x for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:06:15.141 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:14 smithi150 ceph-mon[25582]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:15.142 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:14 smithi150 podman[45783]: 2024-02-10 06:06:14.982884075 +0000 UTC m=+0.114119493 container create b3cac4d40883350d7ba84d57e21e8974e6a66997876626cc457f3c9cb1e1c91b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-10T06:06:15.142 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:14 smithi150 podman[45783]: 2024-02-10 06:06:14.887774378 +0000 UTC m=+0.019009754 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:06:15.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:14 smithi139 ceph-mon[24727]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:15.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:14 smithi139 ceph-mon[29453]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:15.403 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:15 smithi150 podman[45783]: 2024-02-10 06:06:15.235820156 +0000 UTC m=+0.367055528 container init b3cac4d40883350d7ba84d57e21e8974e6a66997876626cc457f3c9cb1e1c91b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.license=GPLv2) 2024-02-10T06:06:15.404 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:15 smithi150 podman[45783]: 2024-02-10 06:06:15.238483934 +0000 UTC m=+0.369719311 container start b3cac4d40883350d7ba84d57e21e8974e6a66997876626cc457f3c9cb1e1c91b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-10T06:06:15.404 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:15 smithi150 ceph-mgr[45803]: -- 172.21.15.150:0/3681101137 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5599131701a0 con 0x559913146c00 2024-02-10T06:06:15.404 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:15 smithi150 bash[45783]: b3cac4d40883350d7ba84d57e21e8974e6a66997876626cc457f3c9cb1e1c91b 2024-02-10T06:06:15.404 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:15 smithi150 systemd[1]: Started Ceph mgr.x for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:06:15.935 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:15.831+0000 7f5a8453f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T06:06:15.935 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:15.934+0000 7f5a8453f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T06:06:16.612 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:16 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:16.613 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:16 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:16.613 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:16 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:06:16.613 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:16 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:06:16.613 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:16 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:06:16.613 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:16 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:16.613 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:16 smithi150 ceph-mon[25582]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:16.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:16.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:16.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:06:16.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:06:16.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:06:16.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:16.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[29453]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:16.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:16.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:16.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:06:16.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:06:16.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:06:16.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:16.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:16 smithi139 ceph-mon[24727]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:16.873 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:16.612+0000 7f5a8453f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T06:06:16.873 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:16.717+0000 7f5a8453f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T06:06:17.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:16.872+0000 7f5a8453f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T06:06:17.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:16.976+0000 7f5a8453f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T06:06:17.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:17.066+0000 7f5a8453f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T06:06:17.467 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:17.347+0000 7f5a8453f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T06:06:17.467 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:17.466+0000 7f5a8453f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T06:06:17.866 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:17.566+0000 7f5a8453f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T06:06:17.867 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:17.678+0000 7f5a8453f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T06:06:17.867 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:17.865+0000 7f5a8453f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T06:06:19.261 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:18 smithi150 ceph-mon[25582]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:19.261 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:18.890+0000 7f5a8453f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T06:06:19.262 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:19 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:19.030+0000 7f5a8453f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T06:06:19.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:18 smithi139 ceph-mon[24727]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:18 smithi139 ceph-mon[29453]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:19.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:19 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:19.383+0000 7f5a8453f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T06:06:19.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:19 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:19.475+0000 7f5a8453f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T06:06:20.086 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:19 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:19.674+0000 7f5a8453f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:06:20.086 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:19 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:19.874+0000 7f5a8453f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T06:06:20.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:20 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:20.176+0000 7f5a8453f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:06:20.748 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:20 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:20.438+0000 7f5a8453f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T06:06:20.748 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:20 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:20.674+0000 7f5a8453f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T06:06:21.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:06:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:06:20] "GET /metrics HTTP/1.1" 200 207979 "" "Prometheus/2.33.4" 2024-02-10T06:06:21.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:20 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:21.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:20 smithi139 ceph-mon[24727]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:20 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:21.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:20 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:21.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:20 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:21.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:20 smithi139 ceph-mon[29453]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:21.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:20 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:21.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:20 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:20 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:20 smithi150 ceph-mon[25582]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:20 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:20 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:06:21.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:21.476+0000 7f5a8453f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T06:06:22.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:22.063+0000 7f5a8453f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T06:06:22.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:22.163+0000 7f5a8453f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T06:06:22.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:22.379+0000 7f5a8453f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T06:06:22.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:06:22] ENGINE Bus STARTING 2024-02-10T06:06:22.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: CherryPy Checker: 2024-02-10T06:06:22.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: The Application mounted at '' has an empty config. 2024-02-10T06:06:22.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:06:22.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:06:22] ENGINE Serving on http://:::9283 2024-02-10T06:06:22.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:06:22] ENGINE Bus STARTED 2024-02-10T06:06:23.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[29453]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:23.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[29453]: Standby manager daemon x restarted 2024-02-10T06:06:23.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[29453]: Standby manager daemon x started 2024-02-10T06:06:23.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:06:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:06:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:06:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:06:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[24727]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[24727]: Standby manager daemon x restarted 2024-02-10T06:06:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[24727]: Standby manager daemon x started 2024-02-10T06:06:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:06:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:06:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:06:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:22 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:06:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-mon[25582]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-mon[25582]: Standby manager daemon x restarted 2024-02-10T06:06:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-mon[25582]: Standby manager daemon x started 2024-02-10T06:06:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:06:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:06:23.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:06:23.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:22 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:06:23.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:06:23.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:06:23.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:23.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:06:23.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:23.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:06:24.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:23 smithi139 ceph-mon[24727]: mgrmap e21: y(active, since 6m), standbys: x 2024-02-10T06:06:24.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:23 smithi139 ceph-mon[29453]: mgrmap e21: y(active, since 6m), standbys: x 2024-02-10T06:06:24.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:23 smithi150 ceph-mon[25582]: mgrmap e21: y(active, since 6m), standbys: x 2024-02-10T06:06:25.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:24 smithi139 ceph-mon[24727]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:25.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:24 smithi139 ceph-mon[29453]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:25.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:24 smithi150 ceph-mon[25582]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:27.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:26 smithi139 ceph-mon[24727]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:27.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:26 smithi139 ceph-mon[29453]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:27.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:26 smithi150 ceph-mon[25582]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:28.065 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:06:29.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:28 smithi139 ceph-mon[24727]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:29.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:28 smithi139 ceph-mon[29453]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:28 smithi150 ceph-mon[25582]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:30.996 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:06:30 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:06:30] "GET /metrics HTTP/1.1" 200 207979 "" "Prometheus/2.33.4" 2024-02-10T06:06:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:30 smithi139 ceph-mon[24727]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:30 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:30 smithi139 ceph-mon[29453]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:30 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:30 smithi150 ceph-mon[25582]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:30 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:33.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:32 smithi139 ceph-mon[24727]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:32 smithi139 ceph-mon[29453]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:33 smithi150 ceph-mon[25582]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:33.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:06:33.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:06:33.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:33.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:06:33.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:33.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:06:35.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:34 smithi139 ceph-mon[24727]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:35.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:34 smithi139 ceph-mon[29453]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:35.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:34 smithi150 ceph-mon[25582]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:36 smithi139 ceph-mon[24727]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:37.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:36 smithi139 ceph-mon[29453]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:37.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:36 smithi150 ceph-mon[25582]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:38.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:06:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:38 smithi139 ceph-mon[24727]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:39.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:38 smithi139 ceph-mon[29453]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:39.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:38 smithi150 ceph-mon[25582]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:40.979 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:06:40 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:06:40] "GET /metrics HTTP/1.1" 200 207991 "" "Prometheus/2.33.4" 2024-02-10T06:06:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:40 smithi139 ceph-mon[29453]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:40 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:40 smithi139 ceph-mon[24727]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:40 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:40 smithi150 ceph-mon[25582]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:41.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:40 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:43.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:42 smithi139 ceph-mon[24727]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:43.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:42 smithi139 ceph-mon[29453]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:43.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:42 smithi150 ceph-mon[25582]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:43.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:06:43.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:06:43.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:43.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:06:43.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:43.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:06:45.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:44 smithi139 ceph-mon[29453]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:45.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:44 smithi139 ceph-mon[24727]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:45.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:44 smithi150 ceph-mon[25582]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:47.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:46 smithi139 ceph-mon[29453]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:47.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:46 smithi139 ceph-mon[24727]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:47.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:46 smithi150 ceph-mon[25582]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:48.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:06:49.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:48 smithi139 ceph-mon[29453]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:49.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:48 smithi139 ceph-mon[24727]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:49.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:48 smithi150 ceph-mon[25582]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:06:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:49 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:06:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:06:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:49 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:06:50.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:06:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:49 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:06:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:06:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:49 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:06:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:06:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:49 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:06:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:06:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:49 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:06:50.985 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:06:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:06:50] "GET /metrics HTTP/1.1" 200 207992 "" "Prometheus/2.33.4" 2024-02-10T06:06:51.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:50 smithi139 ceph-mon[29453]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:50 smithi139 ceph-mon[24727]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:50 smithi150 ceph-mon[25582]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:51 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:52.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:51 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:51 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:53.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:52 smithi139 ceph-mon[24727]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:53.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:52 smithi139 ceph-mon[29453]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:52 smithi150 ceph-mon[25582]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:53.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:06:53.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:06:53.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:53.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:06:53.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:06:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:06:53.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:06:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:54 smithi139 ceph-mon[24727]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:55.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:54 smithi139 ceph-mon[29453]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:55.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:54 smithi150 ceph-mon[25582]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:56 smithi139 ceph-mon[24727]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:56 smithi139 ceph-mon[29453]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:56 smithi150 ceph-mon[25582]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:06:58.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:06:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:06:59.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:06:58 smithi139 ceph-mon[24727]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:59.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:06:58 smithi139 ceph-mon[29453]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:06:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:06:58 smithi150 ceph-mon[25582]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:00.996 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:07:00 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:00] "GET /metrics HTTP/1.1" 200 207992 "" "Prometheus/2.33.4" 2024-02-10T06:07:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:00 smithi139 ceph-mon[24727]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:01.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:00 smithi139 ceph-mon[29453]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:00 smithi150 ceph-mon[25582]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:02.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:02 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:02.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:02 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:02 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:03 smithi139 ceph-mon[24727]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:03.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:03 smithi139 ceph-mon[29453]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:03 smithi150 ceph-mon[25582]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:07:03.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:07:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:03.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:07:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:03.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:07:05.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:04 smithi139 ceph-mon[29453]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:05.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:04 smithi139 ceph-mon[24727]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:05.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:04 smithi150 ceph-mon[25582]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:07.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:06 smithi139 ceph-mon[24727]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:07.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:06 smithi139 ceph-mon[29453]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:07.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:06 smithi150 ceph-mon[25582]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:08.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:07:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:07:09.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:08 smithi139 ceph-mon[24727]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:09.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:08 smithi139 ceph-mon[29453]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:09.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:08 smithi150 ceph-mon[25582]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:10.990 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:07:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:10] "GET /metrics HTTP/1.1" 200 207990 "" "Prometheus/2.33.4" 2024-02-10T06:07:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:10 smithi139 ceph-mon[29453]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:10 smithi139 ceph-mon[24727]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:10 smithi150 ceph-mon[25582]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:12.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:11 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:12.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:11 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:11 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:12 smithi139 ceph-mon[24727]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:13.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:12 smithi139 ceph-mon[29453]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:12 smithi150 ceph-mon[25582]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:13.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:07:13.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:07:13.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:13.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:07:13.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:13.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:07:15.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:14 smithi139 ceph-mon[24727]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:15.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:14 smithi139 ceph-mon[29453]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:15.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:14 smithi150 ceph-mon[25582]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:17.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:16 smithi139 ceph-mon[24727]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:17.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:16 smithi139 ceph-mon[29453]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:17.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:16 smithi150 ceph-mon[25582]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:18.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:07:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:07:19.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:18 smithi139 ceph-mon[24727]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:19.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:18 smithi139 ceph-mon[29453]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:18 smithi150 ceph-mon[25582]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:20.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:07:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:20] "GET /metrics HTTP/1.1" 200 207987 "" "Prometheus/2.33.4" 2024-02-10T06:07:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:20 smithi139 ceph-mon[24727]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:20 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:07:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:20 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:07:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:20 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:07:21.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:20 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:07:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:20 smithi139 ceph-mon[29453]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:20 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:07:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:20 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:07:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:20 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:07:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:20 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:07:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:20 smithi150 ceph-mon[25582]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:20 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:07:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:20 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:07:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:20 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:07:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:20 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:07:22.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:21 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:22.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:21 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:21 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:23.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:23 smithi139 ceph-mon[24727]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:23.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:23 smithi139 ceph-mon[29453]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:23 smithi150 ceph-mon[25582]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:23.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:07:23.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:07:23.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:23.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:07:23.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:23.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:07:25.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:24 smithi139 ceph-mon[24727]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:25.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:24 smithi139 ceph-mon[29453]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:25.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:24 smithi150 ceph-mon[25582]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:27.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:26 smithi139 ceph-mon[24727]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:27.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:26 smithi139 ceph-mon[29453]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:27.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:26 smithi150 ceph-mon[25582]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:28.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:07:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:07:29.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:28 smithi139 ceph-mon[24727]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:29.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:28 smithi139 ceph-mon[29453]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:28 smithi150 ceph-mon[25582]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:30.998 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:07:30 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:30] "GET /metrics HTTP/1.1" 200 207987 "" "Prometheus/2.33.4" 2024-02-10T06:07:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:30 smithi139 ceph-mon[24727]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:30 smithi139 ceph-mon[29453]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:30 smithi150 ceph-mon[25582]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:32.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:32 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:32 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:32 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:33.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:33 smithi139 ceph-mon[24727]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:33.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:33 smithi139 ceph-mon[29453]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:33 smithi150 ceph-mon[25582]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:33.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:07:33.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:07:33.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:33.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:07:33.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:33.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:07:35.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:34 smithi139 ceph-mon[24727]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:35.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:34 smithi139 ceph-mon[29453]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:35.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:34 smithi150 ceph-mon[25582]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:36 smithi139 ceph-mon[24727]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:37.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:36 smithi139 ceph-mon[29453]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:37.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:36 smithi150 ceph-mon[25582]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:38.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:07:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:07:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:38 smithi139 ceph-mon[24727]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:39.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:38 smithi139 ceph-mon[29453]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:39.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:38 smithi150 ceph-mon[25582]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:40.998 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:07:40 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:40] "GET /metrics HTTP/1.1" 200 207991 "" "Prometheus/2.33.4" 2024-02-10T06:07:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:40 smithi139 ceph-mon[24727]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:40 smithi139 ceph-mon[29453]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:40 smithi150 ceph-mon[25582]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:42 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:42.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:42 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:42.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:42 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:43.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:43 smithi139 ceph-mon[24727]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:43.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:43 smithi139 ceph-mon[29453]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:43.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:43 smithi150 ceph-mon[25582]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:43.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:07:43.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:07:43.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:43.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:07:43.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:43.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:07:45.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:44 smithi139 ceph-mon[24727]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:45.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:44 smithi139 ceph-mon[29453]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:45.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:44 smithi150 ceph-mon[25582]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:47.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:46 smithi139 ceph-mon[24727]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:47.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:46 smithi139 ceph-mon[29453]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:47.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:46 smithi150 ceph-mon[25582]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:48.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:07:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:07:49.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:48 smithi139 ceph-mon[24727]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:49.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:48 smithi139 ceph-mon[29453]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:49.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:49 smithi150 ceph-mon[25582]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:07:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:07:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:50 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:07:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:50 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:07:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:07:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:07:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:50 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:07:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:50 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:07:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:07:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:07:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:50 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:07:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:50 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:07:51.007 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:07:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:50] "GET /metrics HTTP/1.1" 200 207987 "" "Prometheus/2.33.4" 2024-02-10T06:07:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:51 smithi139 ceph-mon[24727]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:51.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:51 smithi139 ceph-mon[29453]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:51 smithi150 ceph-mon[25582]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:52 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:52.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:52 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:52 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:53.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:53 smithi139 ceph-mon[24727]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:53.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:53 smithi139 ceph-mon[29453]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:53 smithi150 ceph-mon[25582]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:53.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:07:53.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:07:53.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:07:53.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:07:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:07:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:07:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:54 smithi139 ceph-mon[24727]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:55.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:54 smithi139 ceph-mon[29453]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:55.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:54 smithi150 ceph-mon[25582]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:57 smithi139 ceph-mon[24727]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:57.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:57 smithi139 ceph-mon[29453]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:57 smithi150 ceph-mon[25582]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:07:58.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:07:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:07:59.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:07:59 smithi139 ceph-mon[24727]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:59.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:07:59 smithi139 ceph-mon[29453]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:07:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:07:59 smithi150 ceph-mon[25582]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:01.006 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:00 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:00] "GET /metrics HTTP/1.1" 200 207987 "" "Prometheus/2.33.4" 2024-02-10T06:08:01.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:01 smithi139 ceph-mon[29453]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:01 smithi139 ceph-mon[24727]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:01 smithi150 ceph-mon[25582]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:02.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:02 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:02.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:02 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:02 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:03 smithi139 ceph-mon[24727]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:03.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:03 smithi139 ceph-mon[29453]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:03 smithi150 ceph-mon[25582]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:08:03.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:08:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:03.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:08:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:03.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:08:05.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:04 smithi139 ceph-mon[24727]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:05.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:04 smithi139 ceph-mon[29453]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:05.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:04 smithi150 ceph-mon[25582]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:07.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:07 smithi139 ceph-mon[24727]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:07.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:07 smithi139 ceph-mon[29453]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:07.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:07 smithi150 ceph-mon[25582]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:08.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:08:09.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:09 smithi139 ceph-mon[29453]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:09.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:09 smithi139 ceph-mon[24727]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:09.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:09 smithi150 ceph-mon[25582]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:11.008 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:10] "GET /metrics HTTP/1.1" 200 207988 "" "Prometheus/2.33.4" 2024-02-10T06:08:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:11 smithi139 ceph-mon[24727]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:11 smithi139 ceph-mon[29453]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:11 smithi150 ceph-mon[25582]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:12.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:12 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:12.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:12 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:12 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:13 smithi139 ceph-mon[24727]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:13.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:13 smithi139 ceph-mon[29453]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:13 smithi150 ceph-mon[25582]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:13.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:08:13.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:08:13.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:13.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:08:13.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:13.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:08:14.609 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:08:15.162 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:14 smithi139 ceph-mon[24727]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:15.162 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:14 smithi139 ceph-mon[29453]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:15.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:14 smithi150 ceph-mon[25582]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:16.635 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:08:16.635 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (6m) 2m ago 7m 24.1M - ba2b418f427c adb2e2c020a1 2024-02-10T06:08:16.635 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (7m) 116s ago 7m 54.2M - 8.3.5 dad864ee21e9 db9dee5d8d83 2024-02-10T06:08:16.635 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (6m) 2m ago 6m 53.4M - 3.5 e1d6a67b021e cc86a3c37228 2024-02-10T06:08:16.635 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283 running (2m) 116s ago 11m 66.5M - 19.0.0-1361-g290212a1 c328ab9dcfc7 b3cac4d40883 2024-02-10T06:08:16.635 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:9283 running (13m) 2m ago 13m 459M - 17.2.0 e1d6a67b021e 5d5f244aa01f 2024-02-10T06:08:16.635 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (13m) 2m ago 13m 51.9M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (12m) 116s ago 12m 39.8M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (12m) 2m ago 12m 37.9M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (8m) 2m ago 8m 16.4M - 1dbe0e931976 5b38421d9397 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (8m) 116s ago 8m 17.4M - 1dbe0e931976 c79c2778bd87 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (11m) 2m ago 11m 49.5M 4016M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (11m) 2m ago 11m 49.7M 4016M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (10m) 2m ago 10m 48.2M 4016M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (10m) 2m ago 10m 48.3M 4016M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (9m) 116s ago 9m 49.0M 4272M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (9m) 116s ago 9m 49.9M 4272M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (9m) 116s ago 9m 46.4M 4272M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (8m) 116s ago 8m 51.0M 4272M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (6m) 116s ago 8m 67.4M - 514e6a882f6e 81a684e8f18f 2024-02-10T06:08:16.636 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (6m) 2m ago 6m 69.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:08:16.637 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (6m) 116s ago 6m 69.3M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:08:17.317 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:17 smithi139 ceph-mon[24727]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:17.317 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:17 smithi139 ceph-mon[29453]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:17.340 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T06:08:17.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:17 smithi150 ceph-mon[25582]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:18.012 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:08:18.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:18 smithi139 ceph-mon[24727]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:18 smithi139 ceph-mon[29453]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:18.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:18 smithi150 ceph-mon[25582]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:19.060 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:19 smithi139 ceph-mon[24727]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:19.061 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:19 smithi139 ceph-mon[29453]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:19 smithi150 ceph-mon[25582]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:19.403 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:08:19.403 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:08:19.403 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T06:08:19.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:08:19.403 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:08:19.403 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T06:08:19.403 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:08:19.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": {}, 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:08:19.404 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:08:20.106 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph -s' 2024-02-10T06:08:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:20 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1195788762' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:08:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:20 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1195788762' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:08:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:20 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1195788762' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:08:20.718 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:20] "GET /metrics HTTP/1.1" 200 207980 "" "Prometheus/2.33.4" 2024-02-10T06:08:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:21 smithi150 ceph-mon[25582]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:21 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:21 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:21.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:21 smithi150 ceph-mon[25582]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:08:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:21 smithi139 ceph-mon[24727]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:21 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:21 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:21 smithi139 ceph-mon[24727]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:08:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:21 smithi139 ceph-mon[29453]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:21 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:21 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:21 smithi139 ceph-mon[29453]: from='mgr.24334 172.21.15.139:0/1236540094' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:08:22.144 INFO:teuthology.orchestra.run.smithi139.stdout: cluster: 2024-02-10T06:08:22.144 INFO:teuthology.orchestra.run.smithi139.stdout: id: ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:08:22.144 INFO:teuthology.orchestra.run.smithi139.stdout: health: HEALTH_OK 2024-02-10T06:08:22.144 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:08:22.144 INFO:teuthology.orchestra.run.smithi139.stdout: services: 2024-02-10T06:08:22.144 INFO:teuthology.orchestra.run.smithi139.stdout: mon: 3 daemons, quorum a,c,b (age 12m) 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: mgr: y(active, since 8m), standbys: x 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 9m) 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: data: 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: pools: 6 pools, 161 pgs 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: objects: 209 objects, 585 KiB 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: usage: 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: pgs: 161 active+clean 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: io: 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-10T06:08:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:08:22.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:22 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:22.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:22 smithi139 ceph-mon[24727]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:08:22.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:22 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:22.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:22 smithi139 ceph-mon[29453]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:08:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:22 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:22 smithi150 ceph-mon[25582]: from='mgr.24334 ' entity='mgr.y' 2024-02-10T06:08:22.777 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T06:08:23.323 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:23 smithi139 ceph-mon[24727]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:23.324 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:23 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/508157798' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:08:23.324 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:23 smithi139 ceph-mon[29453]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:23.324 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:23 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/508157798' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:08:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:23 smithi150 ceph-mon[25582]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:23 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/508157798' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:08:23.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:08:23.521Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:08:23.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:23.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:08:23.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:23.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:08:24.850 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_OK 2024-02-10T06:08:25.185 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:24 smithi139 ceph-mon[24727]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:25.185 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:24 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3106156589' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:08:25.185 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:24 smithi139 ceph-mon[29453]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:25.185 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:24 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3106156589' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:08:25.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:24 smithi150 ceph-mon[25582]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:25.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:24 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3106156589' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:08:25.508 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-02-10T06:08:27.261 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:27 smithi139 ceph-mon[24727]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:27.261 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:27 smithi139 ceph-mon[29453]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:27.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:27 smithi150 ceph-mon[25582]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:27.525 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:08:28.012 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:27 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:08:28.165 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:28 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/4051706006' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:08:28.165 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:28 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/4051706006' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:08:28.188 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph mgr fail' 2024-02-10T06:08:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:28 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/4051706006' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:08:29.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:29 smithi139 ceph-mon[24727]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:29.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:29 smithi139 ceph-mon[29453]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:29 smithi150 ceph-mon[25582]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:31.017 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:30 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:30] "GET /metrics HTTP/1.1" 200 207980 "" "Prometheus/2.33.4" 2024-02-10T06:08:31.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-mon[24727]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:31.273 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/441543762' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:08:31.274 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-mon[24727]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T06:08:31.274 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-mon[24727]: Standby manager daemon y started 2024-02-10T06:08:31.274 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:31.024+0000 7f31d77d7700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-10T06:08:31.274 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ignoring --setuser ceph since I am not root 2024-02-10T06:08:31.274 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ignoring --setgroup ceph since I am not root 2024-02-10T06:08:31.274 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-mgr[24973]: -- 172.21.15.139:0/3357001746 <== mon.1 v2:172.21.15.139:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fffb268340 con 0x55fffb240800 2024-02-10T06:08:31.274 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-mon[29453]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:31.275 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/441543762' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:08:31.275 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-mon[29453]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T06:08:31.275 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-mon[29453]: Standby manager daemon y started 2024-02-10T06:08:31.283 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:08:31] ENGINE Bus STOPPING 2024-02-10T06:08:31.283 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:08:31] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T06:08:31.283 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:08:31] ENGINE Bus STOPPED 2024-02-10T06:08:31.283 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:08:31] ENGINE Bus STARTING 2024-02-10T06:08:31.283 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:08:31] ENGINE Serving on http://:::9283 2024-02-10T06:08:31.283 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:08:31] ENGINE Bus STARTED 2024-02-10T06:08:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:31 smithi150 ceph-mon[25582]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:31 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/441543762' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:08:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:31 smithi150 ceph-mon[25582]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T06:08:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:31 smithi150 ceph-mon[25582]: Standby manager daemon y started 2024-02-10T06:08:31.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:31.334Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-02-10T06:08:31.681 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'sleep 180' 2024-02-10T06:08:32.004 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:31.586+0000 7fe62c0cb000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T06:08:32.004 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:31 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:31.684+0000 7fe62c0cb000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T06:08:32.276 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/441543762' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: mgrmap e22: x(active, starting, since 0.828044s), standbys: y 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:08:32.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: Manager daemon x is now available 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: Queued rgw.foo for migration 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:32.278 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/441543762' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: mgrmap e22: x(active, starting, since 0.828044s), standbys: y 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:08:32.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: Manager daemon x is now available 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: Queued rgw.foo for migration 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:32.280 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:32.281 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:32.281 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:08:32.281 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:08:32.281 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:08:32.281 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:08:32.281 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.281 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.281 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:08:32.282 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:08:32.282 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:08:32.282 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.282 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/441543762' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: mgrmap e22: x(active, starting, since 0.828044s), standbys: y 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:08:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: Manager daemon x is now available 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: Queued rgw.foo for migration 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:32.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:08:32.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:08:32.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:08:32.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:08:32.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:32 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:32.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:32.321+0000 7fe62c0cb000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T06:08:32.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:32.410+0000 7fe62c0cb000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T06:08:32.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:32.492+0000 7fe62c0cb000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T06:08:32.922 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:32.643+0000 7fe62c0cb000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T06:08:32.922 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:32.724+0000 7fe62c0cb000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T06:08:32.923 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:32.835+0000 7fe62c0cb000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T06:08:33.179 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:32 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:32.921+0000 7fe62c0cb000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T06:08:33.179 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:33.097+0000 7fe62c0cb000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T06:08:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[24727]: Deploying cephadm binary to smithi150 2024-02-10T06:08:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[24727]: Deploying cephadm binary to smithi139 2024-02-10T06:08:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[24727]: mgrmap e23: x(active, since 1.84463s), standbys: y 2024-02-10T06:08:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[24727]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[24727]: [10/Feb/2024:06:08:32] ENGINE Bus STARTING 2024-02-10T06:08:33.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[24727]: [10/Feb/2024:06:08:32] ENGINE Serving on https://172.21.15.150:7150 2024-02-10T06:08:33.180 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[29453]: Deploying cephadm binary to smithi150 2024-02-10T06:08:33.180 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[29453]: Deploying cephadm binary to smithi139 2024-02-10T06:08:33.180 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[29453]: mgrmap e23: x(active, since 1.84463s), standbys: y 2024-02-10T06:08:33.180 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[29453]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:33.181 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[29453]: [10/Feb/2024:06:08:32] ENGINE Bus STARTING 2024-02-10T06:08:33.181 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-mon[29453]: [10/Feb/2024:06:08:32] ENGINE Serving on https://172.21.15.150:7150 2024-02-10T06:08:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:33 smithi150 ceph-mon[25582]: Deploying cephadm binary to smithi150 2024-02-10T06:08:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:33 smithi150 ceph-mon[25582]: Deploying cephadm binary to smithi139 2024-02-10T06:08:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:33 smithi150 ceph-mon[25582]: mgrmap e23: x(active, since 1.84463s), standbys: y 2024-02-10T06:08:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:33 smithi150 ceph-mon[25582]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:33 smithi150 ceph-mon[25582]: [10/Feb/2024:06:08:32] ENGINE Bus STARTING 2024-02-10T06:08:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:33 smithi150 ceph-mon[25582]: [10/Feb/2024:06:08:32] ENGINE Serving on https://172.21.15.150:7150 2024-02-10T06:08:33.467 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:33.186+0000 7fe62c0cb000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T06:08:33.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:08:33.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:08:33.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:33.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-02-10T06:08:33.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:33.528Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:08:34.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:34.236+0000 7fe62c0cb000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T06:08:34.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:34 smithi139 ceph-mon[24727]: [10/Feb/2024:06:08:32] ENGINE Serving on http://172.21.15.150:8765 2024-02-10T06:08:34.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:34 smithi139 ceph-mon[24727]: [10/Feb/2024:06:08:32] ENGINE Bus STARTED 2024-02-10T06:08:34.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:34 smithi139 ceph-mon[24727]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:34 smithi139 ceph-mon[29453]: [10/Feb/2024:06:08:32] ENGINE Serving on http://172.21.15.150:8765 2024-02-10T06:08:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:34 smithi139 ceph-mon[29453]: [10/Feb/2024:06:08:32] ENGINE Bus STARTED 2024-02-10T06:08:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:34 smithi139 ceph-mon[29453]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:34 smithi150 ceph-mon[25582]: [10/Feb/2024:06:08:32] ENGINE Serving on http://172.21.15.150:8765 2024-02-10T06:08:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:34 smithi150 ceph-mon[25582]: [10/Feb/2024:06:08:32] ENGINE Bus STARTED 2024-02-10T06:08:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:34 smithi150 ceph-mon[25582]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:34.651 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:34.369+0000 7fe62c0cb000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T06:08:34.651 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:34.561+0000 7fe62c0cb000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T06:08:35.054 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:34.650+0000 7fe62c0cb000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T06:08:35.054 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:34.787+0000 7fe62c0cb000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:08:35.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:35 smithi139 ceph-mon[24727]: mgrmap e24: x(active, since 3s), standbys: y 2024-02-10T06:08:35.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:35 smithi139 ceph-mon[29453]: mgrmap e24: x(active, since 3s), standbys: y 2024-02-10T06:08:35.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:35 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:35.115+0000 7fe62c0cb000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T06:08:35.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:35 smithi150 ceph-mon[25582]: mgrmap e24: x(active, since 3s), standbys: y 2024-02-10T06:08:35.597 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:35 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:35.337+0000 7fe62c0cb000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:08:35.597 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:35 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:35.424+0000 7fe62c0cb000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T06:08:36.061 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:35 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:35.595+0000 7fe62c0cb000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T06:08:36.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:36 smithi139 ceph-mon[24727]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:36.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:36 smithi139 ceph-mon[29453]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:36.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:36 smithi150 ceph-mon[25582]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:36.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:36 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:36.553+0000 7fe62c0cb000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T06:08:37.142 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:37 smithi139 ceph-mon[24727]: mgrmap e25: x(active, since 5s), standbys: y 2024-02-10T06:08:37.143 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:36 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:36.875+0000 7fe62c0cb000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T06:08:37.143 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:36 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:36.979+0000 7fe62c0cb000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T06:08:37.143 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:37 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:37.141+0000 7fe62c0cb000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T06:08:37.143 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:37 smithi139 ceph-mon[29453]: mgrmap e25: x(active, since 5s), standbys: y 2024-02-10T06:08:37.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:37 smithi150 ceph-mon[25582]: mgrmap e25: x(active, since 5s), standbys: y 2024-02-10T06:08:37.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:37 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:06:08:37] ENGINE Bus STARTING 2024-02-10T06:08:37.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:37 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: CherryPy Checker: 2024-02-10T06:08:37.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:37 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: The Application mounted at '' has an empty config. 2024-02-10T06:08:37.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:37 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: 2024-02-10T06:08:37.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:37 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:06:08:37] ENGINE Serving on http://:::9283 2024-02-10T06:08:37.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:37 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: [10/Feb/2024:06:08:37] ENGINE Bus STARTED 2024-02-10T06:08:37.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:37] "GET /metrics HTTP/1.1" 200 35107 "" "Prometheus/2.33.4" 2024-02-10T06:08:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[24727]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[24727]: Standby manager daemon y restarted 2024-02-10T06:08:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[24727]: Standby manager daemon y started 2024-02-10T06:08:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T06:08:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:08:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T06:08:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:08:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[29453]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[29453]: Standby manager daemon y restarted 2024-02-10T06:08:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[29453]: Standby manager daemon y started 2024-02-10T06:08:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T06:08:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:08:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T06:08:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:08:38.347 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:38 smithi150 ceph-mon[25582]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:38.347 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:38 smithi150 ceph-mon[25582]: Standby manager daemon y restarted 2024-02-10T06:08:38.347 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:38 smithi150 ceph-mon[25582]: Standby manager daemon y started 2024-02-10T06:08:38.347 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:38 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T06:08:38.347 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:38 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:08:38.348 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:38 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T06:08:38.348 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:38 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3449678741' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:08:39.026 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:08:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:08:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: |\n OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked down and back up at {{ $value | humanize }} times once a minute for 5 minutes. This could indicate a network issue (latency, packet drop, disruption) on the clusters \"cluster network\". Check the network environment on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSD's to flap (mark each other out)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.139:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:08:39.026 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:08:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:08:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephPGImbalance\nexpr: abs(((ceph_osd_numpg > 0) - on(job) group_left() avg by(job) (ceph_osd_numpg\n > 0)) / on(job) group_left() avg by(job) (ceph_osd_numpg > 0)) * on(ceph_daemon)\n group_left(hostname) ceph_osd_metadata > 0.3\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.5\n severity: warning\n type: ceph_default\nannotations:\n description: |\n OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} deviates by more than 30% from average PG count.\n summary: PG allocations are not balanced across devices\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.139:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:08:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:39 smithi139 ceph-mon[24727]: mgrmap e26: x(active, since 7s), standbys: y 2024-02-10T06:08:39.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:39 smithi139 ceph-mon[29453]: mgrmap e26: x(active, since 7s), standbys: y 2024-02-10T06:08:39.337 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:38.946Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:08:39.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:39 smithi150 ceph-mon[25582]: mgrmap e26: x(active, since 7s), standbys: y 2024-02-10T06:08:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:40 smithi139 ceph-mon[24727]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:08:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:40 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:40 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:40.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:40 smithi139 ceph-mon[29453]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:08:40.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:40 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:40.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:40 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:40 smithi150 ceph-mon[25582]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:08:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:40 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:40 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:40 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:08:41.470 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:41.470 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.470 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:41 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:41.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:41.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:41.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:41 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: Adjusting osd_memory_target on smithi139 to 2992M 2024-02-10T06:08:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:08:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: Adjusting osd_memory_target on smithi150 to 3248M 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:08:42.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: Adjusting osd_memory_target on smithi139 to 2992M 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: Adjusting osd_memory_target on smithi150 to 3248M 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:42 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: Adjusting osd_memory_target on smithi139 to 2992M 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: Adjusting osd_memory_target on smithi150 to 3248M 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:42.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:08:42.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:08:42.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:08:42.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:42.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:42 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.086 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:42 smithi139 systemd[1]: Stopping Ceph node-exporter.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:08:43.086 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:42 smithi139 bash[58629]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter.a" found: no such container 2024-02-10T06:08:43.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:42 smithi139 podman[58641]: 2024-02-10 06:08:42.955024653 +0000 UTC m=+0.116269954 container died 5b38421d9397c150faab6ef5b0cb4d663013713a6b6c63b4727f1e28f5178e85 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:43.523 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:43 smithi139 podman[58641]: 2024-02-10 06:08:43.311337804 +0000 UTC m=+0.472583101 container cleanup 5b38421d9397c150faab6ef5b0cb4d663013713a6b6c63b4727f1e28f5178e85 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:43.523 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:43 smithi139 bash[58641]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a 2024-02-10T06:08:43.523 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:43 smithi139 bash[58667]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter.a" found: no such container 2024-02-10T06:08:43.523 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[24727]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:08:43.523 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[24727]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:08:43.524 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:08:43.524 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:08:43.524 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[24727]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:08:43.524 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[24727]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:08:43.524 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.524 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.524 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.524 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.524 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[29453]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:08:43.524 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[29453]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:08:43.524 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:08:43.525 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:08:43.525 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[29453]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:08:43.525 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[29453]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:08:43.525 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.525 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.525 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.525 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:43 smithi150 ceph-mon[25582]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:08:43.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:43 smithi150 ceph-mon[25582]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:08:43.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:43 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:08:43.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:43 smithi150 ceph-mon[25582]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:08:43.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:43 smithi150 ceph-mon[25582]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:08:43.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:43 smithi150 ceph-mon[25582]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:08:43.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:43 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:43 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:43 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:43 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:43.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:43 smithi139 podman[58655]: 2024-02-10 06:08:43.612715757 +0000 UTC m=+0.657734947 container remove 5b38421d9397c150faab6ef5b0cb4d663013713a6b6c63b4727f1e28f5178e85 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:43.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:43 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:08:43.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:43 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T06:08:43.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:43 smithi139 systemd[1]: Stopped Ceph node-exporter.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:08:43.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:43 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Consumed 3.768s CPU time. 2024-02-10T06:08:43.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:43 smithi139 systemd[1]: Starting Ceph node-exporter.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:08:43.837 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=error ts=2024-02-10T06:08:43.523Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:08:43.837 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:43.525Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.139:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.139 because it doesn't contain any IP SANs" 2024-02-10T06:08:43.837 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=warn ts=2024-02-10T06:08:43.525Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.150:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.150 because it doesn't contain any IP SANs" 2024-02-10T06:08:44.141 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:43 smithi139 podman[58736]: 2024-02-10 06:08:43.744054206 +0000 UTC m=+0.017323000 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-10T06:08:44.141 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:43 smithi139 podman[58736]: 2024-02-10 06:08:43.863840417 +0000 UTC m=+0.137109190 container create f2db304d3077a5819956062b83fda3d3979d9cc9e64577114eef00eaee8190d4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:44.141 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 podman[58736]: 2024-02-10 06:08:44.05771095 +0000 UTC m=+0.330979736 container init f2db304d3077a5819956062b83fda3d3979d9cc9e64577114eef00eaee8190d4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:44.141 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 podman[58736]: 2024-02-10 06:08:44.060179984 +0000 UTC m=+0.333448760 container start f2db304d3077a5819956062b83fda3d3979d9cc9e64577114eef00eaee8190d4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:44.141 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.063Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T06:08:44.141 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.063Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T06:08:44.141 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.063Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T06:08:44.141 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.063Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T06:08:44.142 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T06:08:44.142 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T06:08:44.142 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T06:08:44.142 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T06:08:44.142 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T06:08:44.142 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T06:08:44.142 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T06:08:44.142 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T06:08:44.143 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T06:08:44.144 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[58750]: ts=2024-02-10T06:08:44.064Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T06:08:44.145 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 bash[58736]: f2db304d3077a5819956062b83fda3d3979d9cc9e64577114eef00eaee8190d4 2024-02-10T06:08:44.586 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:08:44 smithi139 systemd[1]: Started Ceph node-exporter.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:08:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-mon[24727]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-10T06:08:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-mon[24727]: Reconfiguring daemon node-exporter.a on smithi139 2024-02-10T06:08:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-mon[24727]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:08:44.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:44.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-mon[29453]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-10T06:08:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-mon[29453]: Reconfiguring daemon node-exporter.a on smithi139 2024-02-10T06:08:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-mon[29453]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:08:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:44 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:44 smithi150 ceph-mon[25582]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-10T06:08:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:44 smithi150 ceph-mon[25582]: Reconfiguring daemon node-exporter.a on smithi139 2024-02-10T06:08:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:44 smithi150 ceph-mon[25582]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:08:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:44 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:44.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:44 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:45.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:45 smithi139 ceph-mon[24727]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T06:08:45.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:45 smithi139 ceph-mon[24727]: Reconfiguring daemon alertmanager.a on smithi139 2024-02-10T06:08:45.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:45 smithi139 ceph-mon[29453]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T06:08:45.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:45 smithi139 ceph-mon[29453]: Reconfiguring daemon alertmanager.a on smithi139 2024-02-10T06:08:45.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:45 smithi150 ceph-mon[25582]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T06:08:45.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:45 smithi150 ceph-mon[25582]: Reconfiguring daemon alertmanager.a on smithi139 2024-02-10T06:08:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:46 smithi139 ceph-mon[24727]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:08:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:46 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:46.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:46 smithi139 ceph-mon[29453]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:08:46.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:46 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:46 smithi150 ceph-mon[25582]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:08:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:46 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:48.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:47] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-02-10T06:08:48.384 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:48 smithi150 ceph-mon[25582]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:08:48.552 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:48 smithi139 ceph-mon[24727]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:08:48.552 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:48 smithi139 ceph-mon[29453]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:08:50.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:49 smithi139 ceph-mon[24727]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:08:50.181 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:49 smithi139 ceph-mon[29453]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:08:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:49 smithi150 ceph-mon[25582]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:08:50.927 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:08:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:08:51.195 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:50 smithi139 ceph-mon[24727]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:51.195 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:50 smithi139 ceph-mon[29453]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:51.195 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:51 smithi139 systemd[1]: Stopping Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:08:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:50 smithi150 ceph-mon[25582]: from='client.14829 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:51.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:51 smithi139 bash[59006]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T06:08:51.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:51 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[49429]: level=info ts=2024-02-10T06:08:51.321Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:08:51.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:51 smithi139 podman[59017]: 2024-02-10 06:08:51.341963995 +0000 UTC m=+0.137985155 container died adb2e2c020a12217ed58259c8a77e36f60ded6946a537ce4f89888a99901d534 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:51.931 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:51 smithi139 podman[59017]: 2024-02-10 06:08:51.687371775 +0000 UTC m=+0.483392945 container cleanup adb2e2c020a12217ed58259c8a77e36f60ded6946a537ce4f89888a99901d534 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:51.931 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:51 smithi139 bash[59017]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T06:08:51.931 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:51 smithi139 bash[59043]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T06:08:52.304 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:51 smithi139 ceph-mon[29453]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:52.305 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:51 smithi139 ceph-mon[24727]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:52.305 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:51 smithi139 podman[59031]: 2024-02-10 06:08:51.988836957 +0000 UTC m=+0.647241421 container remove adb2e2c020a12217ed58259c8a77e36f60ded6946a537ce4f89888a99901d534 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:52.305 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 podman[59031]: 2024-02-10 06:08:52.064699498 +0000 UTC m=+0.723103973 volume remove bd138c9d3208bb8540ff7bdcc3812ac52c0c00fcc5ac197b974c1604ab3b0eb6 2024-02-10T06:08:52.305 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T06:08:52.305 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 systemd[1]: Stopped Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:08:52.305 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Consumed 1.049s CPU time. 2024-02-10T06:08:52.305 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 systemd[1]: Starting Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:08:52.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:51 smithi150 ceph-mon[25582]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:52.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 podman[59117]: 2024-02-10 06:08:52.207541481 +0000 UTC m=+0.018338895 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-10T06:08:52.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 podman[59117]: 2024-02-10 06:08:52.357132028 +0000 UTC m=+0.167929429 volume create 93e03c5c68b1eba78c9403fddb28c2a5c0a93320ec025b6ce54411a3088fa1e9 2024-02-10T06:08:52.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 podman[59117]: 2024-02-10 06:08:52.457707586 +0000 UTC m=+0.268504986 container create 5d80a4f13ac5cc64dc74459a8e7676a8f7bfd7fec3c9fb8445ddb40722517003 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:52.624 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:08:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:08:52.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.150:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.139:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:08:53.086 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 podman[59117]: 2024-02-10 06:08:52.743940275 +0000 UTC m=+0.554737682 container init 5d80a4f13ac5cc64dc74459a8e7676a8f7bfd7fec3c9fb8445ddb40722517003 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:53.086 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 podman[59117]: 2024-02-10 06:08:52.7466788 +0000 UTC m=+0.557476214 container start 5d80a4f13ac5cc64dc74459a8e7676a8f7bfd7fec3c9fb8445ddb40722517003 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:08:53.086 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[59133]: level=info ts=2024-02-10T06:08:52.756Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-10T06:08:53.086 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[59133]: level=info ts=2024-02-10T06:08:52.756Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-10T06:08:53.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[59133]: level=info ts=2024-02-10T06:08:52.757Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.139 port=9094 2024-02-10T06:08:53.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[59133]: level=info ts=2024-02-10T06:08:52.758Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T06:08:53.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[59133]: level=info ts=2024-02-10T06:08:52.784Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T06:08:53.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[59133]: level=info ts=2024-02-10T06:08:52.785Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T06:08:53.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[59133]: level=info ts=2024-02-10T06:08:52.787Z caller=main.go:518 msg=Listening address=:9093 2024-02-10T06:08:53.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[59133]: level=info ts=2024-02-10T06:08:52.787Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-10T06:08:53.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 bash[59117]: 5d80a4f13ac5cc64dc74459a8e7676a8f7bfd7fec3c9fb8445ddb40722517003 2024-02-10T06:08:53.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:52 smithi139 systemd[1]: Started Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:08:53.993 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:53.993 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:53.993 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[24727]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:08:53.993 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:08:53.993 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:08:53.993 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:53.993 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[24727]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:08:53.993 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[24727]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:53.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:53.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:53.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[29453]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:08:53.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:08:53.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:08:53.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:53.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[29453]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:08:53.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:53 smithi139 ceph-mon[29453]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:53 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:54.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:53 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:08:54.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:53 smithi150 ceph-mon[25582]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:08:54.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:53 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:08:54.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:53 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:08:54.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:53 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:54.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:53 smithi150 ceph-mon[25582]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:08:54.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:53 smithi150 ceph-mon[25582]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:08:55.086 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:08:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[59133]: level=info ts=2024-02-10T06:08:54.759Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001027508s 2024-02-10T06:08:56.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:56 smithi139 ceph-mon[24727]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:56.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:56 smithi139 ceph-mon[29453]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:56.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-mon[25582]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:56.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:08:56.271+0000 7f5a0a1f6700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:08:56.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Creating ceph-iscsi config... 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Traceback (most recent call last): 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: return _run_code(code, main_globals, None, 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: exec(code, run_globals) 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:08:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:08:56.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:08:56.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:08:56.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:08:56.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:56.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:56.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Traceback (most recent call last): 2024-02-10T06:08:56.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: return future.result(timeout) 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: return self.__get_result() 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: raise self._exception 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: config_blobs=daemon_spec.final_config, 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:08:56.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Creating ceph-iscsi config... 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Traceback (most recent call last): 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: return _run_code(code, main_globals, None, 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: exec(code, run_globals) 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:08:56.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:08:56.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:08:56.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:08:56.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:08:56.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:08:56.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:56.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:57 smithi150 ceph-mon[25582]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:08:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:08:57.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: return future.result(timeout) 2024-02-10T06:08:57.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.__get_result() 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: raise self._exception 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:57 smithi150 ceph-mon[25582]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:08:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:57 smithi139 ceph-mon[24727]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:08:57.587 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:08:57.588 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: return future.result(timeout) 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.__get_result() 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: raise self._exception 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:08:57.589 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:08:57.590 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:57 smithi139 ceph-mon[24727]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:57 smithi139 ceph-mon[29453]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:08:57.591 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:08:57.592 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: return future.result(timeout) 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.__get_result() 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: raise self._exception 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:08:57.593 INFO:journalctl@ceph.mon.c.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:08:57.594 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:08:57.595 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:08:57.595 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:08:57.595 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:08:57.595 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:08:57.595 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:57 smithi139 ceph-mon[29453]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:08:58.098 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:08:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:08:57] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-02-10T06:08:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:58 smithi150 ceph-mon[25582]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:08:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:58 smithi150 ceph-mon[25582]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:58.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:58 smithi139 ceph-mon[24727]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:08:58.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:58 smithi139 ceph-mon[24727]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:08:58.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:58 smithi139 ceph-mon[29453]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:08:58.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:58 smithi139 ceph-mon[29453]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:08:59 smithi139 ceph-mon[24727]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:00.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:08:59 smithi139 ceph-mon[29453]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:08:59 smithi150 ceph-mon[25582]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:01.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:09:00 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[24969]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:09:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:09:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:02 smithi150 ceph-mon[25582]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:09:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:02 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:02.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:02.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.150:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.139:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:09:02.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:02 smithi139 ceph-mon[29453]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:09:02.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:02 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:02.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:02 smithi139 ceph-mon[24727]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:09:02.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:02 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:03.086 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:09:02 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[59133]: level=info ts=2024-02-10T06:09:02.762Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003395097s 2024-02-10T06:09:04.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:04 smithi150 ceph-mon[25582]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:09:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:04 smithi139 ceph-mon[24727]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:09:04.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:04 smithi139 ceph-mon[29453]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:09:05.535 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 systemd[1]: Stopping Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:09:05.535 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 bash[49003]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T06:09:05.819 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.535Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:09:05.819 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.535Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T06:09:05.819 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.535Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T06:09:05.819 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.535Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T06:09:05.820 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.535Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T06:09:05.820 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.535Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T06:09:05.820 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.535Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T06:09:05.820 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.535Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T06:09:05.820 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.535Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T06:09:05.820 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.536Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T06:09:05.820 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.536Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T06:09:05.820 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[42506]: ts=2024-02-10T06:09:05.536Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T06:09:05.820 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 podman[49017]: 2024-02-10 06:09:05.54556373 +0000 UTC m=+0.107498423 container died 81a684e8f18fa525dffca8ce2161701f4b6cc91d6dc0bddeeb1d6e15950ab24b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:09:06.099 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 podman[49017]: 2024-02-10 06:09:05.90924437 +0000 UTC m=+0.471179062 container cleanup 81a684e8f18fa525dffca8ce2161701f4b6cc91d6dc0bddeeb1d6e15950ab24b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:09:06.099 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 bash[49017]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T06:09:06.099 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:05 smithi150 bash[49044]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T06:09:06.349 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:06 smithi139 ceph-mon[24727]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:09:06.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:06 smithi139 ceph-mon[29453]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:09:06.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-mon[25582]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:09:06.374 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 podman[49031]: 2024-02-10 06:09:06.261149516 +0000 UTC m=+0.713177474 container remove 81a684e8f18fa525dffca8ce2161701f4b6cc91d6dc0bddeeb1d6e15950ab24b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:09:06.374 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-10T06:09:06.374 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 systemd[1]: Stopped Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:09:06.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Consumed 2.010s CPU time. 2024-02-10T06:09:06.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 systemd[1]: Starting Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:09:06.773 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 podman[49116]: 2024-02-10 06:09:06.495602554 +0000 UTC m=+0.108402457 container create af676a0ee55f16a9d294ce8c75f4dcf17dc777aa2b9ad4e8844b3bd58e02da9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:09:06.773 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 podman[49116]: 2024-02-10 06:09:06.406409801 +0000 UTC m=+0.019209661 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-10T06:09:06.773 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 podman[49116]: 2024-02-10 06:09:06.681425783 +0000 UTC m=+0.294225643 container init af676a0ee55f16a9d294ce8c75f4dcf17dc777aa2b9ad4e8844b3bd58e02da9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:09:06.773 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 podman[49116]: 2024-02-10 06:09:06.684031834 +0000 UTC m=+0.296831694 container start af676a0ee55f16a9d294ce8c75f4dcf17dc777aa2b9ad4e8844b3bd58e02da9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:09:06.773 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.707Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-10T06:09:06.773 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.707Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-10T06:09:06.773 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.707Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-10T06:09:06.774 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.707Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi150 (none))" 2024-02-10T06:09:06.774 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.707Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T06:09:06.774 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.707Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T06:09:06.774 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.709Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T06:09:06.774 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.709Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-10T06:09:06.774 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.710Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-10T06:09:06.774 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.711Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T06:09:06.774 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.711Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.055µs 2024-02-10T06:09:06.774 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.712Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T06:09:06.774 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 bash[49116]: af676a0ee55f16a9d294ce8c75f4dcf17dc777aa2b9ad4e8844b3bd58e02da9f 2024-02-10T06:09:07.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 systemd[1]: Started Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:09:07.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:06.914Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-02-10T06:09:07.462 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:07 smithi150 systemd[1]: Stopping Ceph node-exporter.b for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:09:07.462 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:07 smithi150 bash[49241]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter.b" found: no such container 2024-02-10T06:09:07.462 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:07.170Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-02-10T06:09:07.462 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:07.170Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-02-10T06:09:07.462 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:07.170Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=45.994µs wal_replay_duration=458.393131ms total_replay_duration=458.459044ms 2024-02-10T06:09:07.462 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:07.174Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T06:09:07.462 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:07.174Z caller=main.go:947 level=info msg="TSDB started" 2024-02-10T06:09:07.462 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:07.174Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T06:09:07.462 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:07.198Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.117186ms db_storage=837ns remote_storage=1.835µs web_handler=397ns query_engine=781ns scrape=562.153µs scrape_sd=179.255µs notify=18.646µs notify_sd=16.513µs rules=21.98517ms 2024-02-10T06:09:07.462 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:07.198Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-10T06:09:07.805 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:07 smithi150 podman[49254]: 2024-02-10 06:09:07.461862355 +0000 UTC m=+0.094066042 container died c79c2778bd87546a8060707c1b8a89b4d8711c3404a139264c4f2feff593567e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:09:08.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:08.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:08.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[24727]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-10T06:09:08.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[24727]: Reconfiguring daemon node-exporter.b on smithi150 2024-02-10T06:09:08.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[24727]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:09:08.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/101079742' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:09:08.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:08.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:08.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[29453]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-10T06:09:08.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[29453]: Reconfiguring daemon node-exporter.b on smithi150 2024-02-10T06:09:08.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[29453]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:09:08.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:07 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/101079742' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:09:08.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:08.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:08.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-mon[25582]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-10T06:09:08.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-mon[25582]: Reconfiguring daemon node-exporter.b on smithi150 2024-02-10T06:09:08.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-mon[25582]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:09:08.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:07 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/101079742' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:09:08.125 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:07 smithi150 podman[49254]: 2024-02-10 06:09:07.945997796 +0000 UTC m=+0.578201480 container cleanup c79c2778bd87546a8060707c1b8a89b4d8711c3404a139264c4f2feff593567e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:09:08.125 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:07 smithi150 bash[49254]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b 2024-02-10T06:09:08.125 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:07 smithi150 bash[49285]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter.b" found: no such container 2024-02-10T06:09:08.459 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 podman[49268]: 2024-02-10 06:09:08.23075361 +0000 UTC m=+0.773646172 container remove c79c2778bd87546a8060707c1b8a89b4d8711c3404a139264c4f2feff593567e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:09:08.459 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:09:08.459 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T06:09:08.459 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 systemd[1]: Stopped Ceph node-exporter.b for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:09:08.459 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Consumed 3.858s CPU time. 2024-02-10T06:09:08.459 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 systemd[1]: Starting Ceph node-exporter.b for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:09:08.767 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 podman[49355]: 2024-02-10 06:09:08.363509759 +0000 UTC m=+0.019998125 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-10T06:09:08.767 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 podman[49355]: 2024-02-10 06:09:08.485526842 +0000 UTC m=+0.142015208 container create 0758bcdcf70aa02fcc554c33740fc7eb07f9173a58ca18c2bcaf7bc5302e7431 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:09:08.767 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 podman[49355]: 2024-02-10 06:09:08.692522283 +0000 UTC m=+0.349010636 container init 0758bcdcf70aa02fcc554c33740fc7eb07f9173a58ca18c2bcaf7bc5302e7431 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:09:08.767 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 podman[49355]: 2024-02-10 06:09:08.695236599 +0000 UTC m=+0.351724953 container start 0758bcdcf70aa02fcc554c33740fc7eb07f9173a58ca18c2bcaf7bc5302e7431 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.698Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.698Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.699Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.699Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.699Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.699Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.699Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.699Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T06:09:08.768 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T06:09:08.769 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T06:09:08.769 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T06:09:08.769 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T06:09:08.769 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T06:09:08.769 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T06:09:08.769 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T06:09:08.769 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T06:09:08.769 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T06:09:08.769 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T06:09:08.770 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T06:09:08.770 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T06:09:08.770 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T06:09:08.770 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T06:09:08.770 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T06:09:08.770 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T06:09:08.770 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T06:09:08.770 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T06:09:08.771 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T06:09:08.772 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T06:09:08.772 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T06:09:08.772 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T06:09:08.772 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T06:09:08.772 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T06:09:08.772 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T06:09:08.772 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[49371]: ts=2024-02-10T06:09:08.700Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T06:09:08.772 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 bash[49355]: 0758bcdcf70aa02fcc554c33740fc7eb07f9173a58ca18c2bcaf7bc5302e7431 2024-02-10T06:09:09.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:08 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2307739763' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1583722991"}]: dispatch 2024-02-10T06:09:09.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:08 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1583722991"}]: dispatch 2024-02-10T06:09:09.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:08 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:09.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:08 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:09.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:08 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2307739763' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1583722991"}]: dispatch 2024-02-10T06:09:09.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:08 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1583722991"}]: dispatch 2024-02-10T06:09:09.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:08 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:09.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:08 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:09.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2307739763' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1583722991"}]: dispatch 2024-02-10T06:09:09.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1583722991"}]: dispatch 2024-02-10T06:09:09.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:09.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:08 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:09.127 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:09:08 smithi150 systemd[1]: Started Ceph node-exporter.b for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:09:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1583722991"}]': finished 2024-02-10T06:09:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[24727]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T06:09:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[24727]: Reconfiguring grafana.a (dependencies changed)... 2024-02-10T06:09:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[24727]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T06:09:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:09:10.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:09:10.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[24727]: Reconfiguring daemon grafana.a on smithi150 2024-02-10T06:09:10.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[24727]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:10.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2113831398' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1583722991"}]: dispatch 2024-02-10T06:09:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1583722991"}]': finished 2024-02-10T06:09:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[29453]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T06:09:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[29453]: Reconfiguring grafana.a (dependencies changed)... 2024-02-10T06:09:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[29453]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T06:09:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:09:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:09:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[29453]: Reconfiguring daemon grafana.a on smithi150 2024-02-10T06:09:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[29453]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:09 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2113831398' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1583722991"}]: dispatch 2024-02-10T06:09:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:09 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/1583722991"}]': finished 2024-02-10T06:09:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:09 smithi150 ceph-mon[25582]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T06:09:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:09 smithi150 ceph-mon[25582]: Reconfiguring grafana.a (dependencies changed)... 2024-02-10T06:09:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:09 smithi150 ceph-mon[25582]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T06:09:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:09 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:09 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:09 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:09:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:09 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:09:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:09 smithi150 ceph-mon[25582]: Reconfiguring daemon grafana.a on smithi150 2024-02-10T06:09:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:09 smithi150 ceph-mon[25582]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:09 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2113831398' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1583722991"}]: dispatch 2024-02-10T06:09:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:10 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2113831398' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1583722991"}]': finished 2024-02-10T06:09:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:10 smithi139 ceph-mon[24727]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T06:09:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:10 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/4247586049' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1839384734"}]: dispatch 2024-02-10T06:09:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:10 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1839384734"}]: dispatch 2024-02-10T06:09:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:10 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2113831398' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1583722991"}]': finished 2024-02-10T06:09:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:10 smithi139 ceph-mon[29453]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T06:09:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:10 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/4247586049' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1839384734"}]: dispatch 2024-02-10T06:09:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:10 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1839384734"}]: dispatch 2024-02-10T06:09:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:10 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2113831398' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/1583722991"}]': finished 2024-02-10T06:09:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:10 smithi150 ceph-mon[25582]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T06:09:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:10 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/4247586049' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1839384734"}]: dispatch 2024-02-10T06:09:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:10 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1839384734"}]: dispatch 2024-02-10T06:09:12.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:11 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1839384734"}]': finished 2024-02-10T06:09:12.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:11 smithi139 ceph-mon[29453]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T06:09:12.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:11 smithi139 ceph-mon[29453]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:12.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:11 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3701537054' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2377362831"}]: dispatch 2024-02-10T06:09:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:11 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1839384734"}]': finished 2024-02-10T06:09:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:11 smithi139 ceph-mon[24727]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T06:09:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:11 smithi139 ceph-mon[24727]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:11 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3701537054' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2377362831"}]: dispatch 2024-02-10T06:09:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:11 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1839384734"}]': finished 2024-02-10T06:09:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:11 smithi150 ceph-mon[25582]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T06:09:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:11 smithi150 ceph-mon[25582]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:11 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3701537054' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2377362831"}]: dispatch 2024-02-10T06:09:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:12 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3701537054' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2377362831"}]': finished 2024-02-10T06:09:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:12 smithi139 ceph-mon[24727]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T06:09:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:12 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1196571400' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3505229686"}]: dispatch 2024-02-10T06:09:13.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:12 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3701537054' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2377362831"}]': finished 2024-02-10T06:09:13.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:12 smithi139 ceph-mon[29453]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T06:09:13.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:12 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1196571400' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3505229686"}]: dispatch 2024-02-10T06:09:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:12 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3701537054' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2377362831"}]': finished 2024-02-10T06:09:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:12 smithi150 ceph-mon[25582]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T06:09:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:12 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1196571400' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3505229686"}]: dispatch 2024-02-10T06:09:14.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:14 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1196571400' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3505229686"}]': finished 2024-02-10T06:09:14.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:14 smithi139 ceph-mon[29453]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T06:09:14.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:14 smithi139 ceph-mon[29453]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:14.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:14 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2100550143' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3021323360"}]: dispatch 2024-02-10T06:09:14.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:14 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1196571400' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3505229686"}]': finished 2024-02-10T06:09:14.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:14 smithi139 ceph-mon[24727]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T06:09:14.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:14 smithi139 ceph-mon[24727]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:14.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:14 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2100550143' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3021323360"}]: dispatch 2024-02-10T06:09:14.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:14 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1196571400' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3505229686"}]': finished 2024-02-10T06:09:14.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:14 smithi150 ceph-mon[25582]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T06:09:14.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:14 smithi150 ceph-mon[25582]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:14.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:14 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2100550143' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3021323360"}]: dispatch 2024-02-10T06:09:15.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:15 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2100550143' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3021323360"}]': finished 2024-02-10T06:09:15.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:15 smithi139 ceph-mon[24727]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T06:09:15.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:15 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3682028711' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/150105896"}]: dispatch 2024-02-10T06:09:15.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:15 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/150105896"}]: dispatch 2024-02-10T06:09:15.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:15 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2100550143' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3021323360"}]': finished 2024-02-10T06:09:15.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:15 smithi139 ceph-mon[29453]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T06:09:15.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:15 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3682028711' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/150105896"}]: dispatch 2024-02-10T06:09:15.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:15 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/150105896"}]: dispatch 2024-02-10T06:09:15.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:15 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2100550143' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3021323360"}]': finished 2024-02-10T06:09:15.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:15 smithi150 ceph-mon[25582]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T06:09:15.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:15 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3682028711' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/150105896"}]: dispatch 2024-02-10T06:09:15.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:15 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/150105896"}]: dispatch 2024-02-10T06:09:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:16 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/150105896"}]': finished 2024-02-10T06:09:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:16 smithi139 ceph-mon[24727]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T06:09:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:16 smithi139 ceph-mon[24727]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:16.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:16 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/150105896"}]': finished 2024-02-10T06:09:16.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:16 smithi139 ceph-mon[29453]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T06:09:16.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:16 smithi139 ceph-mon[29453]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:16.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:16 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/150105896"}]': finished 2024-02-10T06:09:16.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:16 smithi150 ceph-mon[25582]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T06:09:16.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:16 smithi150 ceph-mon[25582]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:17.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:17 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:17.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:17 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:17.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:17 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:18.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:18 smithi139 ceph-mon[24727]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 1 op/s 2024-02-10T06:09:18.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:18 smithi139 ceph-mon[29453]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 1 op/s 2024-02-10T06:09:18.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:18 smithi150 ceph-mon[25582]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 1 op/s 2024-02-10T06:09:19.286 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:19 smithi150 ceph-mon[25582]: from='client.24985 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:19.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:19 smithi139 ceph-mon[24727]: from='client.24985 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:19.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:19 smithi139 ceph-mon[29453]: from='client.24985 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:20.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:20 smithi139 ceph-mon[29453]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-02-10T06:09:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:20 smithi139 ceph-mon[24727]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-02-10T06:09:20.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:20 smithi150 ceph-mon[25582]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-02-10T06:09:21.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:09:21] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-02-10T06:09:21.624 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 systemd[1]: Stopping Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:09:21.625 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 bash[49562]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana.a" found: no such container 2024-02-10T06:09:21.625 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[41703]: t=2024-02-10T06:09:21+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-10T06:09:21.625 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 podman[49574]: 2024-02-10 06:09:21.397445107 +0000 UTC m=+0.112370547 container died db9dee5d8d836c8eb027e50ecb585d7d1e337ed924fa3a672bd4104436ae8f09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, io.buildah.version=1.24.2, vcs-type=git, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, distribution-scope=public, io.openshift.expose-services=, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container) 2024-02-10T06:09:21.920 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 podman[49574]: 2024-02-10 06:09:21.703263605 +0000 UTC m=+0.418189055 container cleanup db9dee5d8d836c8eb027e50ecb585d7d1e337ed924fa3a672bd4104436ae8f09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, com.redhat.component=ubi8-container, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., architecture=x86_64, distribution-scope=public, release=236.1648460182, description=Ceph Grafana Container, version=8.5, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, maintainer=Paul Cuzner ) 2024-02-10T06:09:21.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 bash[49574]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a 2024-02-10T06:09:21.921 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 bash[49606]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana.a" found: no such container 2024-02-10T06:09:22.172 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-mon[25582]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:09:22.172 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 podman[49588]: 2024-02-10 06:09:21.921033281 +0000 UTC m=+0.522996521 container remove db9dee5d8d836c8eb027e50ecb585d7d1e337ed924fa3a672bd4104436ae8f09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, io.buildah.version=1.24.2) 2024-02-10T06:09:22.172 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Deactivated successfully. 2024-02-10T06:09:22.172 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 systemd[1]: Stopped Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:09:22.172 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Consumed 3.401s CPU time. 2024-02-10T06:09:22.172 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:21 smithi150 systemd[1]: Starting Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:09:22.172 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 podman[49684]: 2024-02-10 06:09:22.052043546 +0000 UTC m=+0.018378465 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-02-10T06:09:22.172 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 podman[49684]: 2024-02-10 06:09:22.172377521 +0000 UTC m=+0.138712433 container create 8b8f541bca1a322ef633037fcd2901c35335521e6066517996bc5f6daf198afe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, name=ubi8, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, release=236.1648460182, io.openshift.expose-services=, maintainer=Paul Cuzner , version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-02-10T06:09:22.431 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 podman[49684]: 2024-02-10 06:09:22.366271475 +0000 UTC m=+0.332606394 container init 8b8f541bca1a322ef633037fcd2901c35335521e6066517996bc5f6daf198afe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, architecture=x86_64, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, maintainer=Paul Cuzner , distribution-scope=public, vendor=Red Hat, Inc., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-10T06:09:22.431 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 podman[49684]: 2024-02-10 06:09:22.368936058 +0000 UTC m=+0.335270977 container start 8b8f541bca1a322ef633037fcd2901c35335521e6066517996bc5f6daf198afe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, com.redhat.component=ubi8-container, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , version=8.5, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, description=Ceph Grafana Container, distribution-scope=public, io.buildah.version=1.24.2) 2024-02-10T06:09:22.431 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-10T06:09:22.431 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-10T06:09:22.431 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-10T06:09:22.431 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-10T06:09:22.431 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T06:09:22.431 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T06:09:22.431 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="App mode production" logger=settings 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=884.459µs 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-10T06:09:22.432 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-10T06:09:22.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:22 smithi139 ceph-mon[24727]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:09:22.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:22 smithi139 ceph-mon[29453]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:09:22.697 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STOPPING 2024-02-10T06:09:22.698 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T06:09:22.698 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STOPPED 2024-02-10T06:09:22.698 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STARTING 2024-02-10T06:09:22.698 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Serving on http://:::9283 2024-02-10T06:09:22.698 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STARTED 2024-02-10T06:09:22.698 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-10T06:09:22.698 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-10T06:09:22.698 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-10T06:09:22.698 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 bash[49684]: 8b8f541bca1a322ef633037fcd2901c35335521e6066517996bc5f6daf198afe 2024-02-10T06:09:22.698 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 systemd[1]: Started Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:09:22.699 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-02-10T06:09:22.699 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-10T06:09:22.699 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-02-10T06:09:22.699 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-10T06:09:22.699 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-10T06:09:22.699 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:09:22+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T06:09:23.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STOPPING 2024-02-10T06:09:23.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T06:09:23.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STOPPED 2024-02-10T06:09:23.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STARTING 2024-02-10T06:09:23.125 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Serving on http://:::9283 2024-02-10T06:09:23.125 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STARTED 2024-02-10T06:09:23.125 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STOPPING 2024-02-10T06:09:23.125 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T06:09:23.125 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STOPPED 2024-02-10T06:09:23.125 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STARTING 2024-02-10T06:09:23.125 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Serving on http://:::9283 2024-02-10T06:09:23.125 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:09:22] ENGINE Bus STARTED 2024-02-10T06:09:23.718 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi139.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi139.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi150.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi150.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.719 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:09:23.720 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:09:23.720 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi150.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:09:23.720 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi150.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:09:23.720 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.720 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:09:23.720 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[24727]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:09:23.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:09:23.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:09:23.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi139.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi139.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi150.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi150.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi150.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi150.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:09:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:23 smithi139 ceph-mon[29453]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:09:23.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:09:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:09:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi139.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:09:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi139.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:09:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:09:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:09:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi150.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:09:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi150.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:09:23.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:09:23.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:09:23.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi150.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:09:23.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi150.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:09:23.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:23.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:09:23.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:23 smithi150 ceph-mon[25582]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:09:26.221 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:26 smithi139 ceph-mon[24727]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-02-10T06:09:26.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:26 smithi139 ceph-mon[29453]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-02-10T06:09:26.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:26 smithi150 ceph-mon[25582]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-02-10T06:09:27.700 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:09:27.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:09:27.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:09:27.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:09:27.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:09:27.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:09:27.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[24727]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:09:27.702 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:09:27.703 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:09:27.703 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:27 smithi139 ceph-mon[29453]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:27.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:09:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:09:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:09:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:09:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:09:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:09:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:09:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:09:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:27 smithi150 ceph-mon[25582]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:29.011 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:28 smithi139 ceph-mon[24727]: from='client.24985 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:29.012 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:28 smithi139 ceph-mon[29453]: from='client.24985 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:29.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:28 smithi150 ceph-mon[25582]: from='client.24985 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:29.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:28 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:09:30.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:29 smithi139 ceph-mon[24727]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:29 smithi139 ceph-mon[29453]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:29 smithi150 ceph-mon[25582]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:30.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:09:29.999+0000 7f5a0a1f6700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:09:30.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Creating ceph-iscsi config... 2024-02-10T06:09:30.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:09:30.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:09:30.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:09:30.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:30.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:30.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Traceback (most recent call last): 2024-02-10T06:09:30.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:09:30.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: return _run_code(code, main_globals, None, 2024-02-10T06:09:30.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:09:30.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: exec(code, run_globals) 2024-02-10T06:09:30.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:09:30.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:09:30.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:09:30.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:09:30.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:09:30.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:09:30.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Traceback (most recent call last): 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: return future.result(timeout) 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:09:30.377 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: return self.__get_result() 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: raise self._exception 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: config_blobs=daemon_spec.final_config, 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Creating ceph-iscsi config... 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:30.378 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: Traceback (most recent call last): 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: return _run_code(code, main_globals, None, 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: exec(code, run_globals) 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:30.379 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.237 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:30 smithi139 ceph-mon[24727]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:09:31.238 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:09:31.239 INFO:journalctl@ceph.mon.a.smithi139.stdout: return future.result(timeout) 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.__get_result() 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: raise self._exception 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:09:31.240 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.241 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:30 smithi139 ceph-mon[29453]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:09:31.242 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:09:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: return future.result(timeout) 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.__get_result() 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: raise self._exception 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:09:31.244 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:09:31.245 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:09:31.246 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:09:31.246 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.246 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:30 smithi150 ceph-mon[25582]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:09:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:09:31.283 INFO:journalctl@ceph.mon.b.smithi150.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: return future.result(timeout) 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.__get_result() 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: raise self._exception 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:09:31.284 INFO:journalctl@ceph.mon.b.smithi150.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:09:31.285 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:09:31.286 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:09:31.286 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:09:31.286 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:09:31.286 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:09:31.286 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:09:31.286 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:09:31.286 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:09:31.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:09:31] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-02-10T06:09:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:31 smithi139 ceph-mon[29453]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:31 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:32.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:31 smithi139 ceph-mon[24727]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:32.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:31 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:31 smithi150 ceph-mon[25582]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:31 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:34 smithi150 ceph-mon[25582]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:09:34.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:34 smithi139 ceph-mon[24727]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:09:34.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:34 smithi139 ceph-mon[29453]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:09:36.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:36 smithi150 ceph-mon[25582]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:09:36.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:36 smithi139 ceph-mon[24727]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:09:36.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:36 smithi139 ceph-mon[29453]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:09:38.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:38 smithi150 ceph-mon[25582]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:09:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:38 smithi139 ceph-mon[24727]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:09:38.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:38 smithi139 ceph-mon[29453]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:09:39.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:09:40.328 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:39 smithi139 ceph-mon[24727]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:09:40.329 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:39 smithi139 ceph-mon[29453]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:09:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:39 smithi150 ceph-mon[25582]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:09:41.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:09:41] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-02-10T06:09:42.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:42 smithi150 ceph-mon[25582]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:09:42.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:42 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2988519376' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:09:42.470 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:42 smithi139 ceph-mon[24727]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:09:42.470 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:42 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2988519376' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:09:42.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:42 smithi139 ceph-mon[29453]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:09:42.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:42 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2988519376' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:09:44.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:44 smithi150 ceph-mon[25582]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T06:09:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:44 smithi139 ceph-mon[24727]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T06:09:44.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:44 smithi139 ceph-mon[29453]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T06:09:46.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:46 smithi150 ceph-mon[25582]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T06:09:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:46 smithi139 ceph-mon[24727]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T06:09:46.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:46 smithi139 ceph-mon[29453]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T06:09:47.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:47 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:47.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:47 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:47.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:47 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:48.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:48 smithi150 ceph-mon[25582]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T06:09:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:48 smithi139 ceph-mon[24727]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T06:09:48.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:48 smithi139 ceph-mon[29453]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T06:09:49.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:09:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:49 smithi139 ceph-mon[24727]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:09:50.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:49 smithi139 ceph-mon[29453]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:09:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:49 smithi150 ceph-mon[25582]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:09:51.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:09:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:09:51] "GET /metrics HTTP/1.1" 200 37895 "" "Prometheus/2.33.4" 2024-02-10T06:09:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:52 smithi150 ceph-mon[25582]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:09:52.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:52 smithi139 ceph-mon[24727]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:09:52.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:52 smithi139 ceph-mon[29453]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:09:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:53 smithi150 ceph-mon[25582]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:53.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:53 smithi139 ceph-mon[24727]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:53.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:53 smithi139 ceph-mon[29453]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:54 smithi150 ceph-mon[25582]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T06:09:54.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:54 smithi139 ceph-mon[24727]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T06:09:54.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:54 smithi139 ceph-mon[29453]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T06:09:56.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:56 smithi150 ceph-mon[25582]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:56 smithi139 ceph-mon[24727]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:56.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:56 smithi139 ceph-mon[29453]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:58 smithi150 ceph-mon[25582]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:58.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:58 smithi139 ceph-mon[24727]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:58.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:58 smithi139 ceph-mon[29453]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:09:59.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:09:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:09:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:10:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:09:59 smithi139 ceph-mon[24727]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:00.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:09:59 smithi139 ceph-mon[29453]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:09:59 smithi150 ceph-mon[25582]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:01.238 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:00 smithi139 ceph-mon[24727]: overall HEALTH_OK 2024-02-10T06:10:01.238 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:00 smithi139 ceph-mon[29453]: overall HEALTH_OK 2024-02-10T06:10:01.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:00 smithi150 ceph-mon[25582]: overall HEALTH_OK 2024-02-10T06:10:01.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:10:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:10:01] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-02-10T06:10:02.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:01 smithi139 ceph-mon[24727]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:02.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:01 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:02.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:01 smithi139 ceph-mon[29453]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:02.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:01 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:01 smithi150 ceph-mon[25582]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:01 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:02 smithi139 ceph-mon[24727]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:03.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:02 smithi139 ceph-mon[29453]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:02 smithi150 ceph-mon[25582]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:04.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:03 smithi139 ceph-mon[24727]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:04.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:03 smithi139 ceph-mon[29453]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:04.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:03 smithi150 ceph-mon[25582]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:06.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:06 smithi150 ceph-mon[25582]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:06 smithi139 ceph-mon[24727]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:06.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:06 smithi139 ceph-mon[29453]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:08.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:08 smithi150 ceph-mon[25582]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:08 smithi139 ceph-mon[24727]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:08 smithi139 ceph-mon[29453]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:09.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:10:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:10:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:10:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:09 smithi139 ceph-mon[24727]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:10.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:09 smithi139 ceph-mon[29453]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:09 smithi150 ceph-mon[25582]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:11.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:10:11 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:10:11] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-02-10T06:10:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:12 smithi150 ceph-mon[25582]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:12.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:12 smithi139 ceph-mon[24727]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:12.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:12 smithi139 ceph-mon[29453]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:13 smithi150 ceph-mon[25582]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:13.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:13 smithi139 ceph-mon[24727]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:13.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:13 smithi139 ceph-mon[29453]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:14.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:14 smithi139 ceph-mon[24727]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:14.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:14 smithi139 ceph-mon[29453]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:14 smithi150 ceph-mon[25582]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:16 smithi139 ceph-mon[24727]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:16.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:16 smithi139 ceph-mon[29453]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:16 smithi150 ceph-mon[25582]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:17.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:17 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:17.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:17 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:17.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:17 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:18.525 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:18 smithi150 ceph-mon[25582]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:18 smithi139 ceph-mon[24727]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:18.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:18 smithi139 ceph-mon[29453]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:18.874 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:10:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:10:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:10:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:19 smithi139 ceph-mon[24727]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:20.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:19 smithi139 ceph-mon[29453]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:19 smithi150 ceph-mon[25582]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:21.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:10:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:10:21] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-02-10T06:10:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:22 smithi150 ceph-mon[25582]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:22.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:22 smithi139 ceph-mon[24727]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:22.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:22 smithi139 ceph-mon[29453]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:23 smithi150 ceph-mon[25582]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:23.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:23 smithi139 ceph-mon[24727]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:23.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:23 smithi139 ceph-mon[29453]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:24.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:24 smithi150 ceph-mon[25582]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:24 smithi139 ceph-mon[24727]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:24.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:24 smithi139 ceph-mon[29453]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:26 smithi139 ceph-mon[24727]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:26.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:26 smithi139 ceph-mon[29453]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:26.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:26 smithi150 ceph-mon[25582]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:28 smithi139 ceph-mon[24727]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:28.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:28 smithi139 ceph-mon[29453]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:28.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:28 smithi150 ceph-mon[25582]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:29.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:10:28 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:10:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:10:30.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:29 smithi139 ceph-mon[24727]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:29 smithi139 ceph-mon[29453]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:29 smithi150 ceph-mon[25582]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:31.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:30 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:31.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:30 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:31.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:30 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:31.533 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:10:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:10:31] "GET /metrics HTTP/1.1" 200 37895 "" "Prometheus/2.33.4" 2024-02-10T06:10:32.162 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:31 smithi139 ceph-mon[24727]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:32.162 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:31 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:32.162 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:31 smithi139 ceph-mon[29453]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:32.163 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:31 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:32.256 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:31 smithi150 ceph-mon[25582]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:32.256 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:31 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:33.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:32 smithi139 ceph-mon[24727]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:32 smithi139 ceph-mon[29453]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:32 smithi150 ceph-mon[25582]: from='client.25051 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:34.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:33 smithi139 ceph-mon[24727]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:34.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:33 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:34.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:33 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:34.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:33 smithi139 ceph-mon[29453]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:34.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:33 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:33 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:33 smithi150 ceph-mon[25582]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:33 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:33 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:35.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:35 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:35.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:35 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:35 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:35 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:35 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:35 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:10:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:35 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:10:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:35 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:35.412 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:35.412 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:35.412 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:35.412 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:35.412 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:35.412 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:10:35.412 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:10:35.412 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:35.413 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:35.413 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:35.413 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:35.413 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:35.413 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:35.413 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:10:35.413 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:10:35.413 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:35 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:36.369 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:36 smithi139 ceph-mon[29453]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:10:36.369 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:36 smithi139 ceph-mon[29453]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:10:36.369 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:36 smithi139 ceph-mon[29453]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:36.369 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:36 smithi139 ceph-mon[24727]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:10:36.369 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:36 smithi139 ceph-mon[24727]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:10:36.369 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:36 smithi139 ceph-mon[24727]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:36.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:36 smithi150 ceph-mon[25582]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:10:36.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:36 smithi150 ceph-mon[25582]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:10:36.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:36 smithi150 ceph-mon[25582]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:38.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:38 smithi150 ceph-mon[25582]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:10:38.406 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:38 smithi139 ceph-mon[24727]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:10:38.406 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:38 smithi139 ceph-mon[29453]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:10:39.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:10:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:10:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:10:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[24727]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T06:10:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:10:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:10:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:10:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:10:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2640196838' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:10:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[29453]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T06:10:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:10:40.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:10:40.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:10:40.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:10:40.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:40.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:40 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2640196838' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:10:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:40 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:40 smithi150 ceph-mon[25582]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T06:10:40.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:40 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:40.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:40 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:10:40.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:40 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:10:40.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:40 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:10:40.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:40 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:10:40.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:40 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:40.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:40 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2640196838' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:10:41.585 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:10:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:10:41] "GET /metrics HTTP/1.1" 200 37895 "" "Prometheus/2.33.4" 2024-02-10T06:10:42.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:42 smithi150 ceph-mon[25582]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T06:10:42.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:42 smithi139 ceph-mon[24727]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T06:10:42.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:42 smithi139 ceph-mon[29453]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T06:10:44.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[29453]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[24727]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:44.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:44.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:43 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:43 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:43 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:43 smithi150 ceph-mon[25582]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:44.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:43 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:43 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:43 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:43 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:43 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:10:46.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:46 smithi150 ceph-mon[25582]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:10:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:46 smithi139 ceph-mon[24727]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:10:46.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:46 smithi139 ceph-mon[29453]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:10:47.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:47 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:47.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:47 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:47.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:47 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:48.525 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:48 smithi150 ceph-mon[25582]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:48 smithi139 ceph-mon[24727]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:48.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:48 smithi139 ceph-mon[29453]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:48.777 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:10:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:10:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:10:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:49 smithi139 ceph-mon[24727]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:10:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:49 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:49 smithi139 ceph-mon[29453]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:10:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:49 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:49 smithi150 ceph-mon[25582]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:10:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:49 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:51.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:10:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:10:51] "GET /metrics HTTP/1.1" 200 37902 "" "Prometheus/2.33.4" 2024-02-10T06:10:52.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:52 smithi139 ceph-mon[29453]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:10:52.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:52 smithi139 ceph-mon[24727]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:10:52.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:52 smithi150 ceph-mon[25582]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:10:54.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:54 smithi139 ceph-mon[29453]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:10:54.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:54 smithi139 ceph-mon[24727]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:10:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:54 smithi150 ceph-mon[25582]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:10:56.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:56 smithi139 ceph-mon[29453]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:56 smithi139 ceph-mon[24727]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:56 smithi150 ceph-mon[25582]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:10:58.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:58 smithi139 ceph-mon[29453]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:58.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:58 smithi139 ceph-mon[24727]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:58 smithi150 ceph-mon[25582]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:59.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:10:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:10:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:11:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:59 smithi150 ceph-mon[25582]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:00.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:10:59 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:59 smithi139 ceph-mon[24727]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:10:59 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:00.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:59 smithi139 ceph-mon[29453]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:00.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:10:59 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:01.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:11:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:11:01] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-02-10T06:11:02.397 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:02 smithi139 ceph-mon[24727]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:02.397 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:02 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:02.398 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:02 smithi139 ceph-mon[29453]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:02.398 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:02 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:02.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:02 smithi150 ceph-mon[25582]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:02.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:02 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:04 smithi139 ceph-mon[24727]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:04.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:04 smithi139 ceph-mon[29453]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:04 smithi150 ceph-mon[25582]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:06 smithi139 ceph-mon[24727]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:06.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:06 smithi139 ceph-mon[29453]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:06 smithi150 ceph-mon[25582]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:08 smithi139 ceph-mon[24727]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:08 smithi139 ceph-mon[29453]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:08 smithi150 ceph-mon[25582]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:09.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:11:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:11:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:11:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:09 smithi139 ceph-mon[24727]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:10.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:09 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:10.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:09 smithi139 ceph-mon[29453]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:10.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:09 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:09 smithi150 ceph-mon[25582]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:09 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:11.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:11:11 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:11:11] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-02-10T06:11:12.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:12 smithi139 ceph-mon[24727]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:12.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:12 smithi139 ceph-mon[29453]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:12.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:12 smithi150 ceph-mon[25582]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:14.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:14 smithi139 ceph-mon[24727]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:14.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:14 smithi139 ceph-mon[29453]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:14 smithi150 ceph-mon[25582]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:16 smithi139 ceph-mon[24727]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:16.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:16 smithi139 ceph-mon[29453]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:16 smithi150 ceph-mon[25582]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:17.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:17 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:17.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:17 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:17.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:17 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:18.525 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:18 smithi150 ceph-mon[25582]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:18 smithi139 ceph-mon[24727]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:18.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:18 smithi139 ceph-mon[29453]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:18.874 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:11:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:11:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:11:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:19 smithi139 ceph-mon[24727]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:19 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:20.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:19 smithi139 ceph-mon[29453]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:20.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:19 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:20.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:19 smithi150 ceph-mon[25582]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:20.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:19 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:21.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:11:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:11:21] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-02-10T06:11:22.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:22 smithi139 ceph-mon[24727]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:22.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:22 smithi139 ceph-mon[29453]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:22 smithi150 ceph-mon[25582]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:24 smithi139 ceph-mon[24727]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:24.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:24 smithi139 ceph-mon[29453]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:24.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:24 smithi150 ceph-mon[25582]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:26 smithi139 ceph-mon[24727]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:26.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:26 smithi139 ceph-mon[29453]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:26.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:26 smithi150 ceph-mon[25582]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:28 smithi139 ceph-mon[24727]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:28.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:28 smithi139 ceph-mon[29453]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:28.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:28 smithi150 ceph-mon[25582]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:29.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:11:28 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:11:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:11:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:29 smithi139 ceph-mon[29453]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:29 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:30.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:29 smithi139 ceph-mon[24727]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:30.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:29 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:29 smithi150 ceph-mon[25582]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:29 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:31.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:11:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:11:31] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-02-10T06:11:32.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:32 smithi139 ceph-mon[24727]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:32.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:32.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:32 smithi139 ceph-mon[29453]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:32.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:32.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:32 smithi150 ceph-mon[25582]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:32.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:34.200 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-10T06:11:34.505 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:34 smithi139 ceph-mon[24727]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:34.505 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:34 smithi139 ceph-mon[29453]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:34.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:34 smithi150 ceph-mon[25582]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:36.556 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:36 smithi139 ceph-mon[24727]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:36.556 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:36 smithi139 ceph-mon[29453]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:36.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:36 smithi150 ceph-mon[25582]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:36.728 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled to redeploy mgr.y on host 'smithi139' 2024-02-10T06:11:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:37 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/11467336' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:11:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:37 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:37 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:37 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:37 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:37.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:37 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/11467336' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:11:37.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:37 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:37.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:37 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:37.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:37 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:37.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:37 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:37.391 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps --refresh' 2024-02-10T06:11:37.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:37 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/11467336' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:11:37.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:37 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:37.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:37 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:37.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:37 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:37.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:37 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:38 smithi139 ceph-mon[24727]: from='client.25096 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:38 smithi139 ceph-mon[24727]: Schedule redeploy daemon mgr.y 2024-02-10T06:11:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:38 smithi139 ceph-mon[24727]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:11:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:38 smithi139 ceph-mon[29453]: from='client.25096 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:38 smithi139 ceph-mon[29453]: Schedule redeploy daemon mgr.y 2024-02-10T06:11:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:38 smithi139 ceph-mon[29453]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:11:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:38 smithi150 ceph-mon[25582]: from='client.25096 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:38.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:38 smithi150 ceph-mon[25582]: Schedule redeploy daemon mgr.y 2024-02-10T06:11:38.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:38 smithi150 ceph-mon[25582]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:11:39.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:11:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:11:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:11:39.931 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:11:39.931 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (2m) 57s ago 10m 15.5M - 0.23.0 ba2b418f427c 5d80a4f13ac5 2024-02-10T06:11:39.931 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (2m) 56s ago 10m 44.9M - 8.3.5 dad864ee21e9 8b8f541bca1a 2024-02-10T06:11:39.931 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (61s) 57s ago 9m 49.8M - 3.5 e1d6a67b021e df85d0a3428e 2024-02-10T06:11:39.931 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283 running (5m) 56s ago 15m 506M - 19.0.0-1361-g290212a1 c328ab9dcfc7 b3cac4d40883 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:9283 running (16m) 57s ago 16m 414M - 17.2.0 e1d6a67b021e 5d5f244aa01f 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (16m) 57s ago 17m 59.2M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (15m) 56s ago 15m 47.6M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (15m) 57s ago 15m 47.5M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (2m) 57s ago 11m 17.9M - 1.3.1 1dbe0e931976 f2db304d3077 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (2m) 56s ago 11m 20.0M - 1.3.1 1dbe0e931976 0758bcdcf70a 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (14m) 57s ago 14m 51.6M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (14m) 57s ago 14m 52.1M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (14m) 57s ago 14m 50.7M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (13m) 57s ago 13m 51.0M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (13m) 56s ago 13m 51.6M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (12m) 56s ago 12m 52.4M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:11:39.932 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (12m) 56s ago 12m 48.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:11:39.933 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (12m) 56s ago 12m 53.5M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:11:39.933 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (2m) 56s ago 11m 62.4M - 2.33.4 514e6a882f6e af676a0ee55f 2024-02-10T06:11:39.933 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (10m) 57s ago 10m 71.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:11:39.933 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (10m) 56s ago 10m 71.0M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:11:40.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:39 smithi139 ceph-mon[24727]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:11:40.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:39 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:40.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:39 smithi139 ceph-mon[29453]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:11:40.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:39 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:39 smithi150 ceph-mon[25582]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:11:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:39 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:41.032 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'sleep 180' 2024-02-10T06:11:41.265 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:40 smithi150 ceph-mon[25582]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:41.265 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:40 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:40 smithi139 ceph-mon[24727]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:40 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:40 smithi139 ceph-mon[29453]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:41.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:40 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:41.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:11:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:11:41] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-02-10T06:11:42.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:41 smithi150 ceph-mon[25582]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:11:42.274 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:41 smithi139 ceph-mon[29453]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:11:42.275 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:41 smithi139 ceph-mon[24727]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:11:44.578 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:44 smithi139 ceph-mon[29453]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-10T06:11:44.578 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:44 smithi139 ceph-mon[24727]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-10T06:11:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:44 smithi150 ceph-mon[25582]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-10T06:11:46.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:45 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:46.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:45 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:46.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:45 smithi139 ceph-mon[24727]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:11:46.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:45 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:46.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:45 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:46.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:45 smithi139 ceph-mon[29453]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:11:46.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:45 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:46.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:45 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:46.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:45 smithi150 ceph-mon[25582]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:11:47.049 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:47.049 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:47.049 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:47.049 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:47.050 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:47.051 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:11:47.051 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:11:47.051 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:11:47.051 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:46 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:47.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:46 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:47.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:46 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:47.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:46 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:47.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:46 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:47.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:46 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:47.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:46 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:47.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:46 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:11:47.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:46 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:11:47.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:46 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:11:47.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:46 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:48.052 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:47 smithi139 ceph-mon[24727]: Deploying daemon mgr.y on smithi139 2024-02-10T06:11:48.052 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:47 smithi139 ceph-mon[24727]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-10T06:11:48.052 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:47 smithi139 ceph-mon[29453]: Deploying daemon mgr.y on smithi139 2024-02-10T06:11:48.052 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:47 smithi139 ceph-mon[29453]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-10T06:11:48.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:47 smithi150 ceph-mon[25582]: Deploying daemon mgr.y on smithi139 2024-02-10T06:11:48.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:47 smithi150 ceph-mon[25582]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-10T06:11:48.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:48 smithi139 systemd[1]: Stopping Ceph mgr.y for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:11:48.827 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:48 smithi139 podman[63713]: 2024-02-10 06:11:48.540740676 +0000 UTC m=+0.142546692 container died 5d5f244aa01fc4fc3432eb8aa01d4788e2a4b9f9d696caed15701c36a02717fc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.openshift.tags=base centos centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, architecture=x86_64, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , RELEASE=HEAD, ceph=True, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T06:11:49.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:48 smithi139 podman[63713]: 2024-02-10 06:11:48.926008053 +0000 UTC m=+0.527814068 container cleanup 5d5f244aa01fc4fc3432eb8aa01d4788e2a4b9f9d696caed15701c36a02717fc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.19.8, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, vendor=Red Hat, Inc., release=754, name=centos-stream, version=8, maintainer=Guillaume Abrioux ) 2024-02-10T06:11:49.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:48 smithi139 bash[63713]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y 2024-02-10T06:11:49.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:11:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:11:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:11:49.493 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:49 smithi139 podman[63727]: 2024-02-10 06:11:49.169071833 +0000 UTC m=+0.630044317 container remove 5d5f244aa01fc4fc3432eb8aa01d4788e2a4b9f9d696caed15701c36a02717fc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.buildah.version=1.19.8, vcs-type=git, RELEASE=HEAD, maintainer=Guillaume Abrioux , distribution-scope=public, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-10T06:11:49.494 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:49 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:11:49.494 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:49 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-02-10T06:11:49.494 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:49 smithi139 systemd[1]: Stopped Ceph mgr.y for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:11:49.494 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:49 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.y.service: Consumed 59.127s CPU time. 2024-02-10T06:11:49.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:49 smithi139 systemd[1]: Starting Ceph mgr.y for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:11:49.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:49 smithi139 podman[63858]: 2024-02-10 06:11:49.609966648 +0000 UTC m=+0.017452979 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:11:49.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:49 smithi139 podman[63858]: 2024-02-10 06:11:49.720009561 +0000 UTC m=+0.127495876 container create 082caf31059201e756d7ca81a17910710cb11ea888413e1679ee6203e5a95cb9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:11:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:49 smithi139 ceph-mon[24727]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-02-10T06:11:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:49 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:50.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:49 smithi139 ceph-mon[29453]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-02-10T06:11:50.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:49 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:50.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:49 smithi139 podman[63858]: 2024-02-10 06:11:49.955629113 +0000 UTC m=+0.363115444 container init 082caf31059201e756d7ca81a17910710cb11ea888413e1679ee6203e5a95cb9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-10T06:11:50.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:49 smithi139 podman[63858]: 2024-02-10 06:11:49.959200673 +0000 UTC m=+0.366686998 container start 082caf31059201e756d7ca81a17910710cb11ea888413e1679ee6203e5a95cb9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:11:50.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:49 smithi139 ceph-mgr[63877]: -- 172.21.15.139:0/2100962808 <== mon.1 v2:172.21.15.139:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5645f41fe1a0 con 0x5645f41d4c00 2024-02-10T06:11:50.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:50 smithi139 bash[63858]: 082caf31059201e756d7ca81a17910710cb11ea888413e1679ee6203e5a95cb9 2024-02-10T06:11:50.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:50 smithi139 systemd[1]: Started Ceph mgr.y for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:11:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:49 smithi150 ceph-mon[25582]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-02-10T06:11:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:49 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:50.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:50.531+0000 7ff4003e8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T06:11:50.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:50 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:50.631+0000 7ff4003e8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T06:11:51.319 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.319 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.319 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.319 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.319 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:51.320 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.320 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.320 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.320 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.320 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:51.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:11:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:11:51] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-02-10T06:11:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:51 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:51 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:51 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:51 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:51 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:51.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:51.318+0000 7ff4003e8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T06:11:51.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:51.412+0000 7ff4003e8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T06:11:51.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:51.565+0000 7ff4003e8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T06:11:51.937 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:51.661+0000 7ff4003e8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T06:11:51.937 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:51 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:51.746+0000 7ff4003e8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T06:11:52.234 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:52 smithi139 ceph-mon[24727]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-02-10T06:11:52.234 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:52.020+0000 7ff4003e8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T06:11:52.234 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:52.132+0000 7ff4003e8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T06:11:52.234 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:52.233+0000 7ff4003e8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T06:11:52.234 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:52 smithi139 ceph-mon[29453]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-02-10T06:11:52.531 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:52.338+0000 7ff4003e8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T06:11:52.531 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:52.530+0000 7ff4003e8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T06:11:52.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:52 smithi150 ceph-mon[25582]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-02-10T06:11:53.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:53.538+0000 7ff4003e8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T06:11:53.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:53.672+0000 7ff4003e8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T06:11:54.327 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:54 smithi139 ceph-mon[24727]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-02-10T06:11:54.328 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:54 smithi139 ceph-mon[29453]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-02-10T06:11:54.328 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:54.024+0000 7ff4003e8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T06:11:54.328 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:54.116+0000 7ff4003e8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T06:11:54.328 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:54.327+0000 7ff4003e8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:11:54.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:54.517+0000 7ff4003e8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T06:11:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:54 smithi150 ceph-mon[25582]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-02-10T06:11:55.078 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:54.814+0000 7ff4003e8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:11:55.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:55.077+0000 7ff4003e8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T06:11:55.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:55.319+0000 7ff4003e8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T06:11:56.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:56.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:56.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:56.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:56.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:56.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[24727]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:56.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:56.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:56.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:56.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:56.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:56.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:55 smithi139 ceph-mon[29453]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:56.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:55 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:56.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:55 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:56.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:55 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:56.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:55 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:56.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:55 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:11:56.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:55 smithi150 ceph-mon[25582]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:11:56.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:56 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:56.135+0000 7ff4003e8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T06:11:57.035 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:56 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:56.710+0000 7ff4003e8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T06:11:57.035 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:56 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:56.811+0000 7ff4003e8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T06:11:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[24727]: Standby manager daemon y restarted 2024-02-10T06:11:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[24727]: Standby manager daemon y started 2024-02-10T06:11:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T06:11:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:11:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T06:11:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:11:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[29453]: Standby manager daemon y restarted 2024-02-10T06:11:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[29453]: Standby manager daemon y started 2024-02-10T06:11:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T06:11:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:11:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T06:11:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:11:57.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:57.034+0000 7ff4003e8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T06:11:57.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:11:57] ENGINE Bus STARTING 2024-02-10T06:11:57.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: CherryPy Checker: 2024-02-10T06:11:57.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: The Application mounted at '' has an empty config. 2024-02-10T06:11:57.338 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:11:57.338 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:11:57] ENGINE Serving on http://:::9283 2024-02-10T06:11:57.338 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:11:57 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:11:57] ENGINE Bus STARTED 2024-02-10T06:11:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:57 smithi150 ceph-mon[25582]: Standby manager daemon y restarted 2024-02-10T06:11:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:57 smithi150 ceph-mon[25582]: Standby manager daemon y started 2024-02-10T06:11:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:57 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T06:11:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:57 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:11:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:57 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T06:11:57.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:57 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:11:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:58 smithi150 ceph-mon[25582]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:58 smithi150 ceph-mon[25582]: mgrmap e27: x(active, since 3m), standbys: y 2024-02-10T06:11:58.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:58 smithi139 ceph-mon[29453]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:58.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:58 smithi139 ceph-mon[29453]: mgrmap e27: x(active, since 3m), standbys: y 2024-02-10T06:11:58.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:58 smithi139 ceph-mon[24727]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:58.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:58 smithi139 ceph-mon[24727]: mgrmap e27: x(active, since 3m), standbys: y 2024-02-10T06:11:59.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:11:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:11:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:12:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:59 smithi139 ceph-mon[24727]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:11:59 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:00.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:59 smithi139 ceph-mon[29453]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:00.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:11:59 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:59 smithi150 ceph-mon[25582]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:11:59 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:01.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:12:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:12:01] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-02-10T06:12:02.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:02 smithi139 ceph-mon[24727]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:02.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:02 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:02.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:02 smithi139 ceph-mon[29453]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:02.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:02 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:02.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:02 smithi150 ceph-mon[25582]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:02.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:02 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:04 smithi139 ceph-mon[24727]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:04.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:04 smithi139 ceph-mon[29453]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:04 smithi150 ceph-mon[25582]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:06 smithi139 ceph-mon[24727]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:06.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:06 smithi139 ceph-mon[29453]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:06 smithi150 ceph-mon[25582]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:08 smithi139 ceph-mon[24727]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:08 smithi139 ceph-mon[29453]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:08 smithi150 ceph-mon[25582]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:09.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:12:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:12:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:12:10.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:09 smithi139 ceph-mon[29453]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:10.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:09 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:09 smithi139 ceph-mon[24727]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:09 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:09 smithi150 ceph-mon[25582]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:09 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:11.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:12:11 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:12:11] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-02-10T06:12:12.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:12 smithi139 ceph-mon[24727]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:12.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:12 smithi139 ceph-mon[29453]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:12.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:12 smithi150 ceph-mon[25582]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:14.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:14 smithi139 ceph-mon[24727]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:14 smithi139 ceph-mon[29453]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:14 smithi150 ceph-mon[25582]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:16 smithi139 ceph-mon[24727]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:16 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:16.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:16 smithi139 ceph-mon[29453]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:16.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:16 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:16 smithi150 ceph-mon[25582]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:16 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:18.427 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:18 smithi150 ceph-mon[25582]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:18 smithi139 ceph-mon[24727]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:18.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:18 smithi139 ceph-mon[29453]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:18.874 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:12:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:12:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:12:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:19 smithi139 ceph-mon[24727]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:19 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:19 smithi139 ceph-mon[29453]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:19 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:19 smithi150 ceph-mon[25582]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:19 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:21.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:12:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:12:21] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-02-10T06:12:22.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:22 smithi139 ceph-mon[24727]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:22.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:22 smithi139 ceph-mon[29453]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:22 smithi150 ceph-mon[25582]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:24 smithi139 ceph-mon[24727]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:24.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:24 smithi139 ceph-mon[29453]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:24.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:24 smithi150 ceph-mon[25582]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:26 smithi139 ceph-mon[24727]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:26.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:26 smithi139 ceph-mon[29453]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:26.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:26 smithi150 ceph-mon[25582]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:28 smithi139 ceph-mon[24727]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:28.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:28 smithi139 ceph-mon[29453]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:28.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:28 smithi150 ceph-mon[25582]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:29.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:12:28 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:12:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:12:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:29 smithi139 ceph-mon[29453]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:29 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:30.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:29 smithi139 ceph-mon[24727]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:30.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:29 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:29 smithi150 ceph-mon[25582]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:29 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:31.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:31 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:31.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:31 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:31.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:31 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:31.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:12:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:12:31] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-02-10T06:12:32.487 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:32 smithi139 ceph-mon[24727]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:32.488 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:32 smithi139 ceph-mon[29453]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:32.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:32 smithi150 ceph-mon[25582]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:34.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:34 smithi139 ceph-mon[24727]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:34.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:34 smithi139 ceph-mon[29453]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:34.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:34 smithi150 ceph-mon[25582]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:36.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:36 smithi139 ceph-mon[24727]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:36.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:36 smithi139 ceph-mon[29453]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:36.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:36 smithi150 ceph-mon[25582]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:38 smithi139 ceph-mon[24727]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:38.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:38 smithi139 ceph-mon[29453]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:38 smithi150 ceph-mon[25582]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:39.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:12:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:12:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:12:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:39 smithi139 ceph-mon[24727]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:39 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:39 smithi139 ceph-mon[29453]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:39 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:39 smithi150 ceph-mon[25582]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:39 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:41.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:12:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:12:41] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-02-10T06:12:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:42 smithi139 ceph-mon[24727]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:42.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:42 smithi139 ceph-mon[29453]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:42.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:42 smithi150 ceph-mon[25582]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:44 smithi139 ceph-mon[24727]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:44.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:44 smithi139 ceph-mon[29453]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:44 smithi150 ceph-mon[25582]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:46 smithi139 ceph-mon[24727]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:46 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:46.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:46 smithi139 ceph-mon[29453]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:46.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:46 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:46 smithi150 ceph-mon[25582]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:46 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:48.426 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:48 smithi150 ceph-mon[25582]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:48 smithi139 ceph-mon[24727]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:48.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:48 smithi139 ceph-mon[29453]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:48.874 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:12:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:12:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:12:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:49 smithi139 ceph-mon[24727]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:49 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:50.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:49 smithi139 ceph-mon[29453]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:49 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:49 smithi150 ceph-mon[25582]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:49 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:51.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:12:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:12:51] "GET /metrics HTTP/1.1" 200 37902 "" "Prometheus/2.33.4" 2024-02-10T06:12:52.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:52 smithi139 ceph-mon[24727]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:52.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:52 smithi139 ceph-mon[29453]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:52.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:52 smithi150 ceph-mon[25582]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:54.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:54 smithi139 ceph-mon[24727]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:54.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:54 smithi139 ceph-mon[29453]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:54 smithi150 ceph-mon[25582]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:55.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:55 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:55.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:55 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:55.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:55 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:56 smithi139 ceph-mon[24727]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:56 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:56 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:56 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:12:56.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:56 smithi139 ceph-mon[29453]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:56.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:56 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:56.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:56 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:56.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:56 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:12:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:56 smithi150 ceph-mon[25582]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:56 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:56 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:56 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:12:58.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:58 smithi139 ceph-mon[24727]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:58.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:58 smithi139 ceph-mon[29453]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:58 smithi150 ceph-mon[25582]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:59.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:12:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:12:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:13:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:59 smithi139 ceph-mon[24727]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:12:59 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:00.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:59 smithi139 ceph-mon[29453]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:00.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:12:59 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:59 smithi150 ceph-mon[25582]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:12:59 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:01.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:01 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:01.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:01 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:01.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:01 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:01.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:13:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:13:01] "GET /metrics HTTP/1.1" 200 37905 "" "Prometheus/2.33.4" 2024-02-10T06:13:02.486 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:02 smithi139 ceph-mon[24727]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:02.487 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:02 smithi139 ceph-mon[29453]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:02.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:02 smithi150 ceph-mon[25582]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:04 smithi139 ceph-mon[24727]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:04.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:04 smithi139 ceph-mon[29453]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:04 smithi150 ceph-mon[25582]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:06 smithi139 ceph-mon[24727]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:06.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:06 smithi139 ceph-mon[29453]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:06 smithi150 ceph-mon[25582]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:08 smithi139 ceph-mon[24727]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:08 smithi139 ceph-mon[29453]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:08 smithi150 ceph-mon[25582]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:09.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:13:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:13:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:13:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:09 smithi139 ceph-mon[24727]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:09 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:10.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:09 smithi139 ceph-mon[29453]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:10.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:09 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:09 smithi150 ceph-mon[25582]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:09 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:11.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:13:11 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:13:11] "GET /metrics HTTP/1.1" 200 37905 "" "Prometheus/2.33.4" 2024-02-10T06:13:12.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:12 smithi139 ceph-mon[24727]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:12.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:12 smithi139 ceph-mon[29453]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:12.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:12 smithi150 ceph-mon[25582]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:14.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:14 smithi139 ceph-mon[24727]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:14.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:14 smithi139 ceph-mon[29453]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:14 smithi150 ceph-mon[25582]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:16 smithi139 ceph-mon[24727]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:16 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:16 smithi139 ceph-mon[29453]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:16 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:16 smithi150 ceph-mon[25582]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:16 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:18 smithi139 ceph-mon[24727]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:18.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:18 smithi139 ceph-mon[29453]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:18 smithi150 ceph-mon[25582]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:19.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:13:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:13:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:13:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:19 smithi139 ceph-mon[24727]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:19 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:19 smithi139 ceph-mon[29453]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:19 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:19 smithi150 ceph-mon[25582]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:19 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:21.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:13:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:13:21] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-02-10T06:13:22.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:22 smithi139 ceph-mon[24727]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:22.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:22 smithi139 ceph-mon[29453]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:22 smithi150 ceph-mon[25582]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:24 smithi139 ceph-mon[24727]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:24.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:24 smithi139 ceph-mon[29453]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:24.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:24 smithi150 ceph-mon[25582]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:26 smithi139 ceph-mon[24727]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:26.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:26 smithi139 ceph-mon[29453]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:26.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:26 smithi150 ceph-mon[25582]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:28 smithi139 ceph-mon[24727]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:28.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:28 smithi139 ceph-mon[29453]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:28.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:28 smithi150 ceph-mon[25582]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:29.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:13:28 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:13:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:13:30.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:29 smithi139 ceph-mon[24727]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:29 smithi139 ceph-mon[29453]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:29 smithi150 ceph-mon[25582]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:31.282 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:30 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:30 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:31.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:30 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:31.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:13:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:13:31] "GET /metrics HTTP/1.1" 200 37905 "" "Prometheus/2.33.4" 2024-02-10T06:13:32.238 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:31 smithi139 ceph-mon[29453]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:32.238 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:31 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:32.238 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:31 smithi139 ceph-mon[24727]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:32.238 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:31 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:31 smithi150 ceph-mon[25582]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:31 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:34.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:34 smithi139 ceph-mon[24727]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:34.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:34 smithi139 ceph-mon[29453]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:34.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:34 smithi150 ceph-mon[25582]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:36.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:36 smithi139 ceph-mon[24727]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:36.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:36 smithi139 ceph-mon[29453]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:36.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:36 smithi150 ceph-mon[25582]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:38 smithi139 ceph-mon[24727]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:38.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:38 smithi139 ceph-mon[29453]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:38 smithi150 ceph-mon[25582]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:39.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:13:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:13:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:13:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:39 smithi139 ceph-mon[24727]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:40.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:39 smithi139 ceph-mon[29453]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:39 smithi150 ceph-mon[25582]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:41.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:40 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:40 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:40 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:41.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:13:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:13:41] "GET /metrics HTTP/1.1" 200 37905 "" "Prometheus/2.33.4" 2024-02-10T06:13:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:41 smithi139 ceph-mon[24727]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:42.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:41 smithi139 ceph-mon[29453]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:42.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:41 smithi150 ceph-mon[25582]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:44 smithi139 ceph-mon[24727]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:44 smithi139 ceph-mon[29453]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:44 smithi150 ceph-mon[25582]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:46 smithi139 ceph-mon[24727]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:46 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:46.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:46 smithi139 ceph-mon[29453]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:46.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:46 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:46 smithi150 ceph-mon[25582]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:46 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:48 smithi139 ceph-mon[24727]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:48.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:48 smithi139 ceph-mon[29453]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:48 smithi150 ceph-mon[25582]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:49.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:13:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:13:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"smithi150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.139\", device_class=\"ssd\", hostname=\"smithi139\", instance=\"172.21.15.150:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.139\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T06:13:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:49 smithi139 ceph-mon[24727]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:50.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:49 smithi139 ceph-mon[29453]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:49 smithi150 ceph-mon[25582]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:51.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:50 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:50 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:51.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:50 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:51.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:13:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:13:51] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-02-10T06:13:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:51 smithi139 ceph-mon[24727]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:52.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:51 smithi139 ceph-mon[29453]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:51 smithi150 ceph-mon[25582]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:54.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:54 smithi139 ceph-mon[24727]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:54.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:54 smithi139 ceph-mon[29453]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:54 smithi150 ceph-mon[25582]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:56 smithi139 ceph-mon[24727]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:56 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:13:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:56 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:13:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:56 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:13:56.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:56 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:13:56.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:56 smithi139 ceph-mon[29453]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:56.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:56 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:13:56.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:56 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:13:56.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:56 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:13:56.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:56 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:13:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:56 smithi150 ceph-mon[25582]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:56 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:13:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:56 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:13:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:56 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:13:56.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:56 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:13:58.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:58 smithi139 ceph-mon[24727]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:58.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:58 smithi139 ceph-mon[29453]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:58 smithi150 ceph-mon[25582]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:13:59 smithi139 ceph-mon[24727]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:00.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:13:59 smithi139 ceph-mon[29453]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:13:59 smithi150 ceph-mon[25582]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:01.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:00 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:00 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:01.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:00 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:01.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:14:01] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-02-10T06:14:02.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:01 smithi139 ceph-mon[24727]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:02.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:01 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:02.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:01 smithi139 ceph-mon[29453]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:02.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:01 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:01 smithi150 ceph-mon[25582]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:01 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:04 smithi139 ceph-mon[24727]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:04.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:04 smithi139 ceph-mon[29453]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:04 smithi150 ceph-mon[25582]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:06 smithi139 ceph-mon[24727]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:06 smithi139 ceph-mon[29453]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:06 smithi150 ceph-mon[25582]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:08 smithi139 ceph-mon[24727]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:08 smithi139 ceph-mon[29453]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:08 smithi150 ceph-mon[25582]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:10.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:09 smithi139 ceph-mon[29453]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:09 smithi139 ceph-mon[24727]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:09 smithi150 ceph-mon[25582]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:11.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:10 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:10 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:11.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:10 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:11.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:11 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:14:11] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-02-10T06:14:12.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:11 smithi139 ceph-mon[24727]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:12.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:11 smithi139 ceph-mon[29453]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:11 smithi150 ceph-mon[25582]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:14.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:14 smithi139 ceph-mon[24727]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:14.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:14 smithi139 ceph-mon[29453]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:14 smithi150 ceph-mon[25582]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:16 smithi139 ceph-mon[24727]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:16.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:16 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:16 smithi139 ceph-mon[29453]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:16 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:16 smithi150 ceph-mon[25582]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:16 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:18 smithi139 ceph-mon[24727]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:18.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:18 smithi139 ceph-mon[29453]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:18 smithi150 ceph-mon[25582]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:19 smithi139 ceph-mon[24727]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:19 smithi139 ceph-mon[29453]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:19 smithi150 ceph-mon[25582]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:21.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:20 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:20 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:20 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:21.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:14:21] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-02-10T06:14:22.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:21 smithi139 ceph-mon[24727]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:22.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:21 smithi139 ceph-mon[29453]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:21 smithi150 ceph-mon[25582]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:24 smithi139 ceph-mon[24727]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:24 smithi139 ceph-mon[29453]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:24.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:24 smithi150 ceph-mon[25582]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:26 smithi139 ceph-mon[24727]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:26.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:26 smithi139 ceph-mon[29453]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:26.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:26 smithi150 ceph-mon[25582]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:28.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:28 smithi139 ceph-mon[29453]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:28 smithi139 ceph-mon[24727]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:28.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:28 smithi150 ceph-mon[25582]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:30.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:30 smithi139 ceph-mon[24727]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:30.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:30 smithi139 ceph-mon[29453]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:30.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:30 smithi150 ceph-mon[25582]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:31.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:31 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:31 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:31 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:31.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:14:31] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-02-10T06:14:32.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:32 smithi139 ceph-mon[24727]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:32.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:32 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:32 smithi139 ceph-mon[29453]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:32 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:32 smithi150 ceph-mon[25582]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:32 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:34.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:34 smithi139 ceph-mon[24727]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:34.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:34 smithi139 ceph-mon[29453]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:34.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:34 smithi150 ceph-mon[25582]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:36.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:36 smithi139 ceph-mon[24727]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:36 smithi139 ceph-mon[29453]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:36.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:36 smithi150 ceph-mon[25582]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:38 smithi139 ceph-mon[24727]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:38.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:38 smithi139 ceph-mon[29453]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:38 smithi150 ceph-mon[25582]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:40 smithi139 ceph-mon[24727]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:40.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:40 smithi139 ceph-mon[29453]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:40 smithi150 ceph-mon[25582]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:41.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:41 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:41.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:41 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:41.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:41 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:41.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:14:41] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-02-10T06:14:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:42 smithi139 ceph-mon[24727]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:42.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:42 smithi139 ceph-mon[29453]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:42.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:42 smithi150 ceph-mon[25582]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:44.065 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:14:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:44 smithi139 ceph-mon[24727]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:44.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:44 smithi139 ceph-mon[29453]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:44 smithi150 ceph-mon[25582]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:46.244 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:14:46.244 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (5m) 2m ago 13m 23.9M - 0.23.0 ba2b418f427c 5d80a4f13ac5 2024-02-10T06:14:46.244 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (5m) 3m ago 13m 49.4M - 8.3.5 dad864ee21e9 8b8f541bca1a 2024-02-10T06:14:46.244 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (4m) 2m ago 12m 50.4M - 3.5 e1d6a67b021e df85d0a3428e 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283 running (8m) 3m ago 18m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 b3cac4d40883 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (2m) 2m ago 19m 68.9M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (19m) 2m ago 20m 61.8M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (18m) 3m ago 18m 49.1M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (18m) 2m ago 18m 49.1M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (6m) 2m ago 14m 17.9M - 1.3.1 1dbe0e931976 f2db304d3077 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (5m) 3m ago 14m 17.2M - 1.3.1 1dbe0e931976 0758bcdcf70a 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (17m) 2m ago 17m 52.4M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (17m) 2m ago 17m 53.0M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (17m) 2m ago 17m 52.1M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (16m) 2m ago 16m 52.2M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:14:46.245 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (16m) 3m ago 16m 52.2M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:14:46.246 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (16m) 3m ago 16m 53.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:14:46.246 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (15m) 3m ago 15m 49.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:14:46.246 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (15m) 3m ago 15m 54.7M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:14:46.246 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (5m) 3m ago 14m 66.8M - 2.33.4 514e6a882f6e af676a0ee55f 2024-02-10T06:14:46.246 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (13m) 2m ago 13m 72.2M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:14:46.246 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (13m) 3m ago 13m 71.2M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:14:46.503 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:46 smithi139 ceph-mon[24727]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:46.503 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:46 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:46.503 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:46 smithi139 ceph-mon[29453]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:46.503 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:46 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:46.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:46 smithi150 ceph-mon[25582]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:46.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:46 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:46.850 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T06:14:47.502 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:47 smithi139 ceph-mon[24727]: from='client.25114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:47.503 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:47 smithi139 ceph-mon[29453]: from='client.25114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:47.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:47 smithi150 ceph-mon[25582]: from='client.25114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:48.472 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:48 smithi139 ceph-mon[24727]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:48.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:48 smithi139 ceph-mon[29453]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:48.526 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:48 smithi150 ceph-mon[25582]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": {}, 2024-02-10T06:14:48.962 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:14:48.963 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:14:48.963 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:14:48.963 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:14:48.963 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T06:14:48.963 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:14:48.963 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:14:48.963 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:14:49.286 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:49 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1154992512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:14:49.287 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:49 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1154992512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:14:49.610 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T06:14:49.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:49 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1154992512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:14:50.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:50 smithi139 ceph-mon[24727]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:50.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:50 smithi139 ceph-mon[29453]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:50.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:50 smithi150 ceph-mon[25582]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:51.421 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:51 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:51.421 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:51 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:51.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:51 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:51.625 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:14:51] "GET /metrics HTTP/1.1" 200 37902 "" "Prometheus/2.33.4" 2024-02-10T06:14:51.806 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_OK 2024-02-10T06:14:52.541 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:52 smithi139 ceph-mon[24727]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:52.541 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:52 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2717970816' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:14:52.542 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:52 smithi139 ceph-mon[29453]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:52.542 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:52 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2717970816' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:14:52.567 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph -s' 2024-02-10T06:14:52.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:52 smithi150 ceph-mon[25582]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:52.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:52 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2717970816' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:14:54.584 INFO:teuthology.orchestra.run.smithi139.stdout: cluster: 2024-02-10T06:14:54.584 INFO:teuthology.orchestra.run.smithi139.stdout: id: ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:14:54.584 INFO:teuthology.orchestra.run.smithi139.stdout: health: HEALTH_OK 2024-02-10T06:14:54.584 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:14:54.584 INFO:teuthology.orchestra.run.smithi139.stdout: services: 2024-02-10T06:14:54.584 INFO:teuthology.orchestra.run.smithi139.stdout: mon: 3 daemons, quorum a,c,b (age 18m) 2024-02-10T06:14:54.584 INFO:teuthology.orchestra.run.smithi139.stdout: mgr: x(active, since 6m), standbys: y 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: osd: 8 osds: 8 up (since 15m), 8 in (since 15m) 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: data: 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: pools: 6 pools, 161 pgs 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: objects: 209 objects, 585 KiB 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: usage: 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: pgs: 161 active+clean 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: io: 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-02-10T06:14:54.585 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:14:54.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:54 smithi139 ceph-mon[24727]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:54.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:54 smithi139 ceph-mon[29453]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:54 smithi150 ceph-mon[25582]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:55.204 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph mgr fail' 2024-02-10T06:14:55.511 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:55 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3299844677' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:14:55.512 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:55 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3299844677' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:14:55.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:55 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3299844677' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:14:56.555 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:56 smithi139 ceph-mon[24727]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:56.555 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:56 smithi139 ceph-mon[29453]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:56.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:56 smithi150 ceph-mon[25582]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:57 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:57 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:57 smithi139 ceph-mon[24727]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:57.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:57 smithi139 ceph-mon[24727]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:14:57.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:57 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:57.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:57 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:57.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:57 smithi139 ceph-mon[29453]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:57.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:57 smithi139 ceph-mon[29453]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:14:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:57 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:57 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:57 smithi150 ceph-mon[25582]: from='mgr.34769 172.21.15.150:0/3362321585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:57.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:57 smithi150 ceph-mon[25582]: from='mgr.34769 ' entity='mgr.x' 2024-02-10T06:14:58.486 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-mon[24727]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:58.486 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1245143207' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:14:58.486 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:14:58.486 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-mon[24727]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T06:14:58.487 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-mon[29453]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:58.487 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1245143207' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:14:58.487 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:14:58.487 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-mon[29453]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T06:14:58.487 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:14:58] ENGINE Bus STOPPING 2024-02-10T06:14:58.487 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:14:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T06:14:58.487 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:14:58] ENGINE Bus STOPPED 2024-02-10T06:14:58.487 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:14:58] ENGINE Bus STARTING 2024-02-10T06:14:58.487 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:14:58] ENGINE Serving on http://:::9283 2024-02-10T06:14:58.487 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:14:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:14:58] ENGINE Bus STARTED 2024-02-10T06:14:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:58 smithi150 ceph-mon[25582]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:58 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1245143207' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:14:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:58 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:14:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:58 smithi150 ceph-mon[25582]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T06:14:58.625 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:14:58.242+0000 7f5a77b3f700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-10T06:14:58.625 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ignoring --setuser ceph since I am not root 2024-02-10T06:14:58.625 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ignoring --setgroup ceph since I am not root 2024-02-10T06:14:58.625 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:58 smithi150 ceph-mgr[45803]: -- 172.21.15.150:0/192622957 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5575eea1c1a0 con 0x5575ee9f3000 2024-02-10T06:14:58.945 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'sleep 180' 2024-02-10T06:14:59.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:14:58.837+0000 7f5f7e80f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T06:14:59.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:14:58.935+0000 7f5f7e80f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T06:14:59.557 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: mgrmap e28: y(active, starting, since 0.918924s) 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:14:59.558 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: Manager daemon y is now available 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: mgrmap e28: y(active, starting, since 0.918924s) 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:14:59.559 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: Manager daemon y is now available 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:59.560 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:59.561 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:14:59.561 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:14:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:14:59.592 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: mgrmap e28: y(active, starting, since 0.918924s) 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:14:59.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:14:59.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:14:59.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:14:59.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:14:59.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: Manager daemon y is now available 2024-02-10T06:14:59.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:59.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:59.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:59.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:14:59.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:14:59.866 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:14:59.627+0000 7f5f7e80f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T06:14:59.866 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:14:59.722+0000 7f5f7e80f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T06:14:59.866 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:14:59.865+0000 7f5f7e80f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T06:15:00.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:14:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:14:59.977+0000 7f5f7e80f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T06:15:00.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:00.060+0000 7f5f7e80f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T06:15:00.310 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[24727]: mgrmap e29: y(active, since 1.9215s) 2024-02-10T06:15:00.311 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[24727]: [10/Feb/2024:06:14:59] ENGINE Bus STARTING 2024-02-10T06:15:00.311 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[24727]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:00.311 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[24727]: [10/Feb/2024:06:14:59] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T06:15:00.311 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[24727]: [10/Feb/2024:06:14:59] ENGINE Serving on http://172.21.15.139:8765 2024-02-10T06:15:00.311 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[24727]: [10/Feb/2024:06:14:59] ENGINE Bus STARTED 2024-02-10T06:15:00.311 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[29453]: mgrmap e29: y(active, since 1.9215s) 2024-02-10T06:15:00.311 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[29453]: [10/Feb/2024:06:14:59] ENGINE Bus STARTING 2024-02-10T06:15:00.311 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[29453]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:00.311 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[29453]: [10/Feb/2024:06:14:59] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T06:15:00.312 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[29453]: [10/Feb/2024:06:14:59] ENGINE Serving on http://172.21.15.139:8765 2024-02-10T06:15:00.312 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:00 smithi139 ceph-mon[29453]: [10/Feb/2024:06:14:59] ENGINE Bus STARTED 2024-02-10T06:15:00.451 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:00.337+0000 7f5f7e80f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T06:15:00.451 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-mon[25582]: mgrmap e29: y(active, since 1.9215s) 2024-02-10T06:15:00.451 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-mon[25582]: [10/Feb/2024:06:14:59] ENGINE Bus STARTING 2024-02-10T06:15:00.451 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-mon[25582]: from='client.15171 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:00.451 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-mon[25582]: [10/Feb/2024:06:14:59] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T06:15:00.451 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-mon[25582]: [10/Feb/2024:06:14:59] ENGINE Serving on http://172.21.15.139:8765 2024-02-10T06:15:00.451 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-mon[25582]: [10/Feb/2024:06:14:59] ENGINE Bus STARTED 2024-02-10T06:15:00.844 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:00.450+0000 7f5f7e80f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T06:15:00.845 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:00.537+0000 7f5f7e80f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T06:15:00.845 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:00.641+0000 7f5f7e80f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T06:15:00.845 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:00.843+0000 7f5f7e80f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T06:15:01.267 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:01 smithi139 ceph-mon[24727]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:01.267 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:01 smithi139 ceph-mon[24727]: mgrmap e30: y(active, since 2s) 2024-02-10T06:15:01.268 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:01 smithi139 ceph-mon[29453]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:01.268 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:01 smithi139 ceph-mon[29453]: mgrmap e30: y(active, since 2s) 2024-02-10T06:15:01.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:01 smithi150 ceph-mon[25582]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:01.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:01 smithi150 ceph-mon[25582]: mgrmap e30: y(active, since 2s) 2024-02-10T06:15:02.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:01.865+0000 7f5f7e80f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T06:15:02.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:02.002+0000 7f5f7e80f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T06:15:02.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:02.356+0000 7f5f7e80f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T06:15:02.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:02.449+0000 7f5f7e80f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T06:15:03.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:02.656+0000 7f5f7e80f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:15:03.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:02.847+0000 7f5f7e80f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T06:15:03.435 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:03 smithi150 ceph-mon[25582]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:03.436 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:03 smithi150 ceph-mon[25582]: mgrmap e31: y(active, since 4s) 2024-02-10T06:15:03.436 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:03 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:03.436 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:03 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:03.436 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:03 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:03.156+0000 7f5f7e80f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:15:03.436 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:03 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:03.435+0000 7f5f7e80f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T06:15:03.564 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:03 smithi139 ceph-mon[24727]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:03.564 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:03 smithi139 ceph-mon[24727]: mgrmap e31: y(active, since 4s) 2024-02-10T06:15:03.564 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:03 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:03.564 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:03 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:03.564 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:03 smithi139 ceph-mon[29453]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:03.565 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:03 smithi139 ceph-mon[29453]: mgrmap e31: y(active, since 4s) 2024-02-10T06:15:03.565 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:03 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:03.565 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:03 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:04.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:03 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:03.687+0000 7f5f7e80f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T06:15:04.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:04.526+0000 7f5f7e80f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T06:15:05.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:05 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:05 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:05 smithi139 ceph-mon[29453]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:05.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:05 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:05 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:05 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:05 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:05 smithi139 ceph-mon[24727]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:05.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:05 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:05 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:05.110+0000 7f5f7e80f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T06:15:05.376 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:05.210+0000 7f5f7e80f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T06:15:05.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-mon[25582]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:05.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:05.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:05.429+0000 7f5f7e80f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T06:15:05.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:15:05] ENGINE Bus STARTING 2024-02-10T06:15:05.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: CherryPy Checker: 2024-02-10T06:15:05.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: The Application mounted at '' has an empty config. 2024-02-10T06:15:05.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:15:05.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:15:05] ENGINE Serving on http://:::9283 2024-02-10T06:15:05.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:15:05] ENGINE Bus STARTED 2024-02-10T06:15:06.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: Standby manager daemon x started 2024-02-10T06:15:06.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:15:06.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:15:06.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:15:06.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:15:06.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:06.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:06.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:06.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:06.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:15:06.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:15:06.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:15:06.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[29453]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:15:06.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: Standby manager daemon x started 2024-02-10T06:15:06.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:15:06.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:15:06.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:15:06.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:15:06.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:06.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:06.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:06.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:06.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:15:06.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:15:06.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:15:06.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:06 smithi139 ceph-mon[24727]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:15:06.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: Standby manager daemon x started 2024-02-10T06:15:06.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:15:06.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:15:06.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:15:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:15:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:15:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:15:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:15:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:06 smithi150 ceph-mon[25582]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:15:07.374 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:07.179Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.150:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.150:8765: connect: connection refused" 2024-02-10T06:15:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:07.179Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.150:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.150:8765: connect: connection refused" 2024-02-10T06:15:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:07.181Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.150:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.150:8765: connect: connection refused" 2024-02-10T06:15:07.375 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:07.182Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.150:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.150:8765: connect: connection refused" 2024-02-10T06:15:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: mgrmap e32: y(active, since 8s), standbys: x 2024-02-10T06:15:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:15:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:15:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:15:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:15:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:15:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:15:07.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:15:07.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:15:07.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:07.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:07 smithi150 ceph-mon[25582]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:15:07.376 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: mgrmap e32: y(active, since 8s), standbys: x 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:15:07.377 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[24727]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: mgrmap e32: y(active, since 8s), standbys: x 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:07.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:15:07.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:15:07.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:07.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:07 smithi139 ceph-mon[29453]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:15:10.030 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-mon[24727]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:15:10.030 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-mon[29453]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:15:10.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:15:10.096+0000 7ff38611f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:15:10.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Creating ceph-iscsi config... 2024-02-10T06:15:10.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return _run_code(code, main_globals, None, 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: exec(code, run_globals) 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:15:10.337 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:15:10.338 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:15:10.338 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:15:10.338 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:15:10.338 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:10.338 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:10.338 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:15:10.338 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:15:10.338 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return future.result(timeout) 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return self.__get_result() 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: raise self._exception 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: config_blobs=daemon_spec.final_config, 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:15:10.339 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Creating ceph-iscsi config... 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return _run_code(code, main_globals, None, 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: exec(code, run_globals) 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:15:10.340 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:15:10.341 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:15:10.341 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:15:10.341 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:15:10.341 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:15:10.341 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:10.341 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:10 smithi150 ceph-mon[25582]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:15:11.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:15:11 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:15:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:15:12.250 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:12 smithi150 ceph-mon[25582]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:15:12.251 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:15:12.252 INFO:journalctl@ceph.mon.b.smithi150.stdout: return future.result(timeout) 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.__get_result() 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: raise self._exception 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.253 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.254 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.255 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:12 smithi150 ceph-mon[25582]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:15:12.255 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:12 smithi150 ceph-mon[25582]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:15:12.255 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:12 smithi150 ceph-mon[25582]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:15:12.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:12 smithi139 ceph-mon[24727]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:15:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:15:12.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: return future.result(timeout) 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.__get_result() 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: raise self._exception 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:15:12.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:15:12.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:15:12.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:15:12.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:15:12.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:15:12.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:15:12.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:15:12.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:15:12.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:15:12.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:15:12.341 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:15:12.341 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:15:12.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:12 smithi139 ceph-mon[29453]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:15:12.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:15:12.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:15:12.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:15:12.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:15:12.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:15:12.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:15:12.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:15:12.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:15:12.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:15:12.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:15:12.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:15:12.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:15:12.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:15:12.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:15:12.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:15:12.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: return future.result(timeout) 2024-02-10T06:15:12.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:15:12.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.__get_result() 2024-02-10T06:15:12.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:15:12.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: raise self._exception 2024-02-10T06:15:12.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:15:12.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:15:12.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:15:12.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:15:12.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:15:12.345 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:15:12.346 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:15:12.346 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:15:12.346 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:15:12.346 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:15:12.346 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.346 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:12 smithi139 ceph-mon[24727]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:15:12.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:12 smithi139 ceph-mon[24727]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:15:12.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:12 smithi139 ceph-mon[24727]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:15:12.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:15:12.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:15:12.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:15:12.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:15:12.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:15:12.347 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:12 smithi139 ceph-mon[29453]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:15:12.347 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:12 smithi139 ceph-mon[29453]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:15:12.347 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:12 smithi139 ceph-mon[29453]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:15:13.694 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 systemd[1]: Stopping Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:15:13.695 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 bash[54628]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T06:15:14.004 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.694Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:15:14.004 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.694Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T06:15:14.004 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.694Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T06:15:14.004 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.694Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T06:15:14.004 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.694Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T06:15:14.004 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.694Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T06:15:14.004 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.694Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T06:15:14.004 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.694Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T06:15:14.004 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.694Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T06:15:14.005 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.695Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T06:15:14.005 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.695Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T06:15:14.005 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[49131]: ts=2024-02-10T06:15:13.695Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T06:15:14.005 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:13 smithi150 podman[54640]: 2024-02-10 06:15:13.70468365 +0000 UTC m=+0.122075403 container died af676a0ee55f16a9d294ce8c75f4dcf17dc777aa2b9ad4e8844b3bd58e02da9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:15:14.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:14 smithi139 ceph-mon[24727]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:15:14.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:14 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:14.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:14 smithi139 ceph-mon[29453]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:15:14.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:14 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:14.353 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 podman[54640]: 2024-02-10 06:15:14.102858948 +0000 UTC m=+0.520250710 container cleanup af676a0ee55f16a9d294ce8c75f4dcf17dc777aa2b9ad4e8844b3bd58e02da9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:15:14.354 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 bash[54640]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T06:15:14.354 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 bash[54667]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T06:15:14.354 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-mon[25582]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:15:14.354 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:14.608 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 podman[54654]: 2024-02-10 06:15:14.354133715 +0000 UTC m=+0.647867369 container remove af676a0ee55f16a9d294ce8c75f4dcf17dc777aa2b9ad4e8844b3bd58e02da9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:15:14.608 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-10T06:15:14.608 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 systemd[1]: Stopped Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:15:14.608 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Consumed 1.896s CPU time. 2024-02-10T06:15:14.608 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 systemd[1]: Starting Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:15:14.608 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 podman[54738]: 2024-02-10 06:15:14.492540098 +0000 UTC m=+0.019826322 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-10T06:15:14.609 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 podman[54738]: 2024-02-10 06:15:14.605207619 +0000 UTC m=+0.132493830 container create 08b74b342924689fd7b68aa35d527add9bc6271fa2f14c5ce77c775621a0bdf2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:15:14.858 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 podman[54738]: 2024-02-10 06:15:14.832706752 +0000 UTC m=+0.359992964 container init 08b74b342924689fd7b68aa35d527add9bc6271fa2f14c5ce77c775621a0bdf2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:15:14.858 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 podman[54738]: 2024-02-10 06:15:14.83522721 +0000 UTC m=+0.362513421 container start 08b74b342924689fd7b68aa35d527add9bc6271fa2f14c5ce77c775621a0bdf2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:15:15.125 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.858Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-10T06:15:15.125 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.858Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-10T06:15:15.125 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.858Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-10T06:15:15.125 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.858Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi150 (none))" 2024-02-10T06:15:15.125 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.858Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T06:15:15.125 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.858Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T06:15:15.126 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.860Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T06:15:15.126 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.860Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-10T06:15:15.126 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.861Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-10T06:15:15.126 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.862Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T06:15:15.126 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.862Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.187µs 2024-02-10T06:15:15.126 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:14.862Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T06:15:15.126 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 bash[54738]: 08b74b342924689fd7b68aa35d527add9bc6271fa2f14c5ce77c775621a0bdf2 2024-02-10T06:15:15.126 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:14 smithi150 systemd[1]: Started Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:15:15.126 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:15.079Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-02-10T06:15:15.628 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:15.326Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-02-10T06:15:15.628 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:15.370Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-02-10T06:15:15.628 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:15.370Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-02-10T06:15:15.628 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:15.370Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=46.855µs wal_replay_duration=507.327611ms total_replay_duration=507.409327ms 2024-02-10T06:15:15.628 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:15.374Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T06:15:15.628 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:15.374Z caller=main.go:947 level=info msg="TSDB started" 2024-02-10T06:15:15.628 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:15.374Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T06:15:15.628 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:15.392Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.226536ms db_storage=889ns remote_storage=1.215µs web_handler=404ns query_engine=680ns scrape=601.038µs scrape_sd=76.195µs notify=11.764µs notify_sd=10.72µs rules=17.190158ms 2024-02-10T06:15:15.629 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:15:15.392Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-10T06:15:16.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-mon[25582]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:15:16.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:16.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:16.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:15:16.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-mon[25582]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:15:16.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:15 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[24727]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:15:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:15:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[24727]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:15:16.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:16.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[29453]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:15:16.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:16.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:16.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:15:16.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[29453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:15:16.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:15 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:18.203 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:17 smithi150 ceph-mon[25582]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:15:18.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:17 smithi139 ceph-mon[24727]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:15:18.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:17 smithi139 ceph-mon[29453]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:15:20.142 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[24727]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:15:20.142 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:20.142 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:20.143 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:20.143 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:20.143 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:20.143 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:15:20.143 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:20.143 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[29453]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:15:20.143 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:20.143 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:20.143 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:20.143 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:20.143 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:20.144 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:15:20.144 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:20 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:20 smithi150 ceph-mon[25582]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:15:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:20 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:20.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:20 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:20.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:20 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:20.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:20 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:20.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:20 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:20.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:20 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:15:20.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:20 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:21.147 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:21 smithi139 ceph-mon[24727]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:15:21.147 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:21 smithi139 ceph-mon[24727]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:15:21.148 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:21 smithi139 ceph-mon[29453]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:15:21.148 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:21 smithi139 ceph-mon[29453]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:15:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:21 smithi150 ceph-mon[25582]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:15:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:21 smithi150 ceph-mon[25582]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:15:22.332 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:22 smithi139 ceph-mon[24727]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:15:22.370 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:22 smithi139 ceph-mon[29453]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:15:22.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:22 smithi150 ceph-mon[25582]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:15:23.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[29453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[29453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[29453]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/855326411' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[24727]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:15:23.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:15:23.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[24727]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:15:23.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:23.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[24727]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:23.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:23 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/855326411' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:15:23.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:23 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:23.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:23 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:23.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:23 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:15:23.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:23 smithi150 ceph-mon[25582]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:15:23.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:23 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:15:23.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:23 smithi150 ceph-mon[25582]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:15:23.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:23 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:23.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:23 smithi150 ceph-mon[25582]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:23.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:23 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/855326411' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:15:26.318 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:26 smithi139 ceph-mon[24727]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:26.319 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:26 smithi139 ceph-mon[29453]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:26.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:26 smithi150 ceph-mon[25582]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:15:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[24727]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:15:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:28.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[29453]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:15:28.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:28.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:28.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:28.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:28.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:27 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:28.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:27 smithi150 ceph-mon[25582]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:15:28.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:27 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:28.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:27 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:28.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:27 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:28.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:27 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:28.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:27 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:15:29.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:28 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:29.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:28 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:29.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:15:28] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.33.4" 2024-02-10T06:15:29.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:28 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:30.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:30 smithi139 ceph-mon[24727]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:15:30.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:30 smithi139 ceph-mon[29453]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:15:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:30 smithi150 ceph-mon[25582]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:15:32.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:32 smithi139 ceph-mon[24727]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:15:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:32 smithi139 ceph-mon[29453]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:15:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:32 smithi150 ceph-mon[25582]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:15:34.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:34 smithi139 ceph-mon[24727]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:15:34.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:34 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:34 smithi139 ceph-mon[29453]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:15:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:34 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:34 smithi150 ceph-mon[25582]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:15:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:34 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:36.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:36 smithi139 ceph-mon[24727]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:15:36.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:36 smithi139 ceph-mon[29453]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:15:36.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:36 smithi150 ceph-mon[25582]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:15:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:38 smithi139 ceph-mon[24727]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:15:38.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:38 smithi139 ceph-mon[29453]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:15:38.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:38 smithi150 ceph-mon[25582]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:15:39.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:15:38] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.33.4" 2024-02-10T06:15:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:40 smithi139 ceph-mon[24727]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:40.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:40 smithi139 ceph-mon[29453]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:40 smithi150 ceph-mon[25582]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:42 smithi139 ceph-mon[24727]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:42.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:42 smithi139 ceph-mon[29453]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:42.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:42 smithi150 ceph-mon[25582]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:44 smithi139 ceph-mon[24727]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:44 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:44 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:44 smithi139 ceph-mon[29453]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:44 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:44 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:44.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:44 smithi150 ceph-mon[25582]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:44.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:44 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:44.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:44 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:46.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:46 smithi139 ceph-mon[24727]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:46.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:46 smithi139 ceph-mon[29453]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:46.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:46 smithi150 ceph-mon[25582]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:48.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:48 smithi139 ceph-mon[24727]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:48.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:48 smithi139 ceph-mon[29453]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:48.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:48 smithi150 ceph-mon[25582]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:49.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:15:48] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-02-10T06:15:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:50 smithi139 ceph-mon[24727]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:50.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:50 smithi139 ceph-mon[29453]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:50 smithi150 ceph-mon[25582]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:51 smithi139 ceph-mon[24727]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:51.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:51 smithi139 ceph-mon[29453]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:51 smithi150 ceph-mon[25582]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:53.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:53 smithi139 ceph-mon[29453]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:53.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:53 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:53.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:53 smithi139 ceph-mon[24727]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:53.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:53 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:53.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:53 smithi150 ceph-mon[25582]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:53.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:53 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:56.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:56 smithi139 ceph-mon[24727]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:56.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:56 smithi139 ceph-mon[29453]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:56.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:56 smithi150 ceph-mon[25582]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:58.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:58 smithi139 ceph-mon[29453]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:58 smithi139 ceph-mon[24727]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:58 smithi150 ceph-mon[25582]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:59.022 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:15:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:15:58] "GET /metrics HTTP/1.1" 200 37912 "" "Prometheus/2.33.4" 2024-02-10T06:15:59.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:15:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:59.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:15:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:15:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:00 smithi139 ceph-mon[24727]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:00.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:00 smithi139 ceph-mon[29453]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:00 smithi150 ceph-mon[25582]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:02.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:02 smithi139 ceph-mon[24727]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:02.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:02 smithi139 ceph-mon[29453]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:02 smithi150 ceph-mon[25582]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:04.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:04 smithi139 ceph-mon[24727]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:04.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:04 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:04.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:04 smithi139 ceph-mon[29453]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:04.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:04 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:04.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:04 smithi150 ceph-mon[25582]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:04.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:04 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:05.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:05 smithi139 ceph-mon[24727]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:05.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:05 smithi139 ceph-mon[29453]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:05.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:05 smithi150 ceph-mon[25582]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:07 smithi139 ceph-mon[24727]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:07 smithi139 ceph-mon[29453]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:07 smithi150 ceph-mon[25582]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:09.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:16:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:16:08] "GET /metrics HTTP/1.1" 200 37912 "" "Prometheus/2.33.4" 2024-02-10T06:16:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:10 smithi139 ceph-mon[24727]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:10.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:10 smithi139 ceph-mon[29453]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:10 smithi150 ceph-mon[25582]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:12.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:12 smithi139 ceph-mon[24727]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:12.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:12 smithi139 ceph-mon[29453]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:12 smithi150 ceph-mon[25582]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:14.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:14 smithi139 ceph-mon[24727]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:14.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:14 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:14.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:14 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:14.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:14 smithi139 ceph-mon[29453]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:14.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:14 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:14.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:14 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:14.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:14 smithi150 ceph-mon[25582]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:14.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:14 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:14.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:14 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:16 smithi139 ceph-mon[24727]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:16.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:16 smithi139 ceph-mon[29453]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:16.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:16 smithi150 ceph-mon[25582]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:18.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:18 smithi139 ceph-mon[24727]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:18.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:18 smithi139 ceph-mon[29453]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:18.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:18 smithi150 ceph-mon[25582]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:19.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:16:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:16:18] "GET /metrics HTTP/1.1" 200 37913 "" "Prometheus/2.33.4" 2024-02-10T06:16:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:20 smithi139 ceph-mon[24727]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:20.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:20 smithi139 ceph-mon[29453]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:20 smithi150 ceph-mon[25582]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:21 smithi139 ceph-mon[24727]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:21 smithi139 ceph-mon[29453]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:21 smithi150 ceph-mon[25582]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:23.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:23 smithi139 ceph-mon[24727]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:23.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:23 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:23.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:23 smithi139 ceph-mon[29453]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:23.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:23 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:23.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:23 smithi150 ceph-mon[25582]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:23.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:23 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:26.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:26 smithi139 ceph-mon[24727]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:26.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:26 smithi139 ceph-mon[29453]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:26.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:26 smithi150 ceph-mon[25582]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:27.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:27 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:16:27.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:27 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:16:27.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:27 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:16:28.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[24727]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:28.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:16:28.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:16:28.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[29453]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:28.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:16:28.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:16:28.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:28 smithi150 ceph-mon[25582]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:28 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:28 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:28 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:28 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:28 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:16:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:28 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:16:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:28 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:16:29.033 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:16:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:16:28] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-02-10T06:16:29.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:29 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:29.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:29 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:29 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:30.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:30 smithi139 ceph-mon[24727]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:30 smithi139 ceph-mon[29453]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:30 smithi150 ceph-mon[25582]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:31 smithi139 ceph-mon[29453]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:31 smithi139 ceph-mon[24727]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:31 smithi150 ceph-mon[25582]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:33.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:33 smithi139 ceph-mon[24727]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:33.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:33 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:33.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:33 smithi139 ceph-mon[29453]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:33.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:33 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:33.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:33 smithi150 ceph-mon[25582]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:33.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:33 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:36.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:36 smithi139 ceph-mon[24727]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:36.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:36 smithi139 ceph-mon[29453]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:36 smithi150 ceph-mon[25582]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:38 smithi139 ceph-mon[24727]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:38 smithi139 ceph-mon[29453]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:38.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:38 smithi150 ceph-mon[25582]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:39.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:16:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:16:38] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-02-10T06:16:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:40 smithi139 ceph-mon[24727]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:40 smithi139 ceph-mon[29453]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:40 smithi150 ceph-mon[25582]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:42 smithi139 ceph-mon[24727]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:42.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:42 smithi139 ceph-mon[29453]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:42.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:42 smithi150 ceph-mon[25582]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:43.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:43 smithi150 ceph-mon[25582]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:43.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:43 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:43.488 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:43 smithi139 ceph-mon[24727]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:43.488 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:43 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:43.488 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:43 smithi139 ceph-mon[29453]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:43.488 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:43 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:44 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:44 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:44.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:44 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:45.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:45 smithi150 ceph-mon[25582]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:45.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:45 smithi139 ceph-mon[24727]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:45.595 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:45 smithi139 ceph-mon[29453]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:47.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:47 smithi139 ceph-mon[24727]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:47.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:47 smithi139 ceph-mon[29453]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:47.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:47 smithi150 ceph-mon[25582]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:49.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:16:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:16:48] "GET /metrics HTTP/1.1" 200 37909 "" "Prometheus/2.33.4" 2024-02-10T06:16:50.328 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:50 smithi150 ceph-mon[25582]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:50 smithi139 ceph-mon[24727]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:50.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:50 smithi139 ceph-mon[29453]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:52 smithi139 ceph-mon[24727]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:52 smithi139 ceph-mon[29453]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:52 smithi150 ceph-mon[25582]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:54 smithi139 ceph-mon[24727]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:54 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:54 smithi139 ceph-mon[29453]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:54 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:54 smithi150 ceph-mon[25582]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:54 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:56.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:56 smithi139 ceph-mon[24727]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:56.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:56 smithi139 ceph-mon[29453]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:56.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:56 smithi150 ceph-mon[25582]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:58.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:58 smithi139 ceph-mon[29453]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:58 smithi139 ceph-mon[24727]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:58 smithi150 ceph-mon[25582]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:59.045 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:16:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:16:58] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-02-10T06:16:59.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:16:59 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:59.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:16:59 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:16:59 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:00 smithi139 ceph-mon[24727]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:00.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:00 smithi139 ceph-mon[29453]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:00 smithi150 ceph-mon[25582]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:01 smithi139 ceph-mon[24727]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:01.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:01 smithi139 ceph-mon[29453]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:01 smithi150 ceph-mon[25582]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:03.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:03 smithi150 ceph-mon[25582]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:03.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:03 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:03.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:03 smithi139 ceph-mon[29453]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:03.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:03 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:03 smithi139 ceph-mon[24727]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:03 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:06.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:06 smithi139 ceph-mon[24727]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:06.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:06 smithi139 ceph-mon[29453]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:06.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:06 smithi150 ceph-mon[25582]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:08.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:08 smithi139 ceph-mon[24727]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:08.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:08 smithi139 ceph-mon[29453]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:08.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:08 smithi150 ceph-mon[25582]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:09.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:17:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:17:08] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-02-10T06:17:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:10 smithi139 ceph-mon[24727]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:10.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:10 smithi139 ceph-mon[29453]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:10 smithi150 ceph-mon[25582]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:11 smithi150 ceph-mon[25582]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:11.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:11 smithi139 ceph-mon[24727]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:11.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:11 smithi139 ceph-mon[29453]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:13 smithi150 ceph-mon[25582]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:13 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:13.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:13 smithi139 ceph-mon[24727]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:13.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:13 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:13.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:13 smithi139 ceph-mon[29453]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:13.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:13 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:14 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:14.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:14 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:14.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:14 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:16 smithi139 ceph-mon[24727]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:16.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:16 smithi139 ceph-mon[29453]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:16.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:16 smithi150 ceph-mon[25582]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:18.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:18 smithi139 ceph-mon[24727]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:18 smithi139 ceph-mon[29453]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:18.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:18 smithi150 ceph-mon[25582]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:19.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:17:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:17:18] "GET /metrics HTTP/1.1" 200 37919 "" "Prometheus/2.33.4" 2024-02-10T06:17:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:20 smithi139 ceph-mon[24727]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:20 smithi139 ceph-mon[29453]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:20 smithi150 ceph-mon[25582]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:22.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:22 smithi139 ceph-mon[24727]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:22.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:22 smithi139 ceph-mon[29453]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:22 smithi150 ceph-mon[25582]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:23 smithi150 ceph-mon[25582]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:23 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:23.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:23 smithi139 ceph-mon[24727]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:23.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:23 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:23.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:23 smithi139 ceph-mon[29453]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:23.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:23 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:26.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:26 smithi139 ceph-mon[24727]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:26.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:26 smithi139 ceph-mon[29453]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:26.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:26 smithi150 ceph-mon[25582]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:28.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:28 smithi139 ceph-mon[24727]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:28.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:28 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:17:28.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:28 smithi139 ceph-mon[29453]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:28.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:28 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:17:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:28 smithi150 ceph-mon[25582]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:28 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:17:29.041 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:17:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:17:28] "GET /metrics HTTP/1.1" 200 37922 "" "Prometheus/2.33.4" 2024-02-10T06:17:29.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:29 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:29.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:29 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:17:29.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:29 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:17:29.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:29 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:17:29.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:29 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:29.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:29 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:17:29.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:29 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:17:29.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:29 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:17:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:29 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:29 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:17:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:29 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:17:29.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:29 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' 2024-02-10T06:17:30.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:30 smithi139 ceph-mon[24727]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:30.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:30 smithi139 ceph-mon[29453]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:30 smithi150 ceph-mon[25582]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:31 smithi139 ceph-mon[24727]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:31 smithi139 ceph-mon[29453]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:31 smithi150 ceph-mon[25582]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:33.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:33 smithi150 ceph-mon[25582]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:33.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:33 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:33.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:33 smithi139 ceph-mon[24727]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:33.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:33 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:33.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:33 smithi139 ceph-mon[29453]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:33.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:33 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:36.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:36 smithi139 ceph-mon[24727]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:36.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:36 smithi139 ceph-mon[29453]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:36.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:36 smithi150 ceph-mon[25582]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:38 smithi139 ceph-mon[24727]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:38 smithi139 ceph-mon[29453]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:38.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:38 smithi150 ceph-mon[25582]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:39.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:17:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:17:38] "GET /metrics HTTP/1.1" 200 37922 "" "Prometheus/2.33.4" 2024-02-10T06:17:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:40 smithi139 ceph-mon[24727]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:40 smithi139 ceph-mon[29453]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:40 smithi150 ceph-mon[25582]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:41 smithi139 ceph-mon[24727]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:41 smithi139 ceph-mon[29453]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:41 smithi150 ceph-mon[25582]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:43.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:43 smithi150 ceph-mon[25582]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:43.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:43 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:43.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:43 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:43.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:43 smithi139 ceph-mon[24727]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:43.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:43 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:43.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:43 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:43.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:43 smithi139 ceph-mon[29453]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:43.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:43 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:43.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:43 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:46.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:46 smithi139 ceph-mon[24727]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:46.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:46 smithi139 ceph-mon[29453]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:46.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:46 smithi150 ceph-mon[25582]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:48.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:48 smithi139 ceph-mon[24727]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:48.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:48 smithi139 ceph-mon[29453]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:48.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:48 smithi150 ceph-mon[25582]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:49.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:17:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:17:48] "GET /metrics HTTP/1.1" 200 37919 "" "Prometheus/2.33.4" 2024-02-10T06:17:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:50 smithi139 ceph-mon[24727]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:50 smithi139 ceph-mon[29453]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:50 smithi150 ceph-mon[25582]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:52 smithi139 ceph-mon[24727]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:52 smithi139 ceph-mon[29453]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:52 smithi150 ceph-mon[25582]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:54 smithi139 ceph-mon[24727]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:54 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:54 smithi139 ceph-mon[29453]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:54 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:54 smithi150 ceph-mon[25582]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:54 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:55.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:55 smithi150 ceph-mon[25582]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:55.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:55 smithi139 ceph-mon[24727]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:55.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:55 smithi139 ceph-mon[29453]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:57 smithi150 ceph-mon[25582]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:57.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:57 smithi139 ceph-mon[24727]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:57.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:57 smithi139 ceph-mon[29453]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:17:58 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:58.659 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:17:58 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:58.659 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:17:58 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:59.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:17:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:17:58] "GET /metrics HTTP/1.1" 200 37911 "" "Prometheus/2.33.4" 2024-02-10T06:18:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:00 smithi139 ceph-mon[24727]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:00.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:00 smithi139 ceph-mon[29453]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:00 smithi150 ceph-mon[25582]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:01 smithi139 ceph-mon[24727]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:01 smithi139 ceph-mon[29453]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:01 smithi150 ceph-mon[25582]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:01.493 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:18:03.509 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (9m) 2m ago 17m 22.1M - 0.23.0 ba2b418f427c 5d80a4f13ac5 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (8m) 2m ago 16m 53.0M - 8.3.5 dad864ee21e9 8b8f541bca1a 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (2m) 2m ago 16m 50.0M - 3.5 e1d6a67b021e 1a76fd5ed7d3 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283 running (11m) 2m ago 21m 447M - 19.0.0-1361-g290212a1 c328ab9dcfc7 b3cac4d40883 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (6m) 2m ago 23m 497M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (23m) 2m ago 23m 67.7M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (21m) 2m ago 21m 47.5M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (22m) 2m ago 22m 47.3M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (9m) 2m ago 17m 17.1M - 1.3.1 1dbe0e931976 f2db304d3077 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (8m) 2m ago 17m 19.4M - 1.3.1 1dbe0e931976 0758bcdcf70a 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (21m) 2m ago 21m 53.0M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (20m) 2m ago 20m 53.5M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (20m) 2m ago 20m 52.5M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (20m) 2m ago 20m 51.7M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:18:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (19m) 2m ago 19m 52.9M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:18:03.511 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (19m) 2m ago 19m 53.9M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:18:03.511 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (19m) 2m ago 18m 50.6M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:18:03.511 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (18m) 2m ago 18m 55.7M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:18:03.511 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (2m) 2m ago 17m 49.1M - 2.33.4 514e6a882f6e 08b74b342924 2024-02-10T06:18:03.511 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (16m) 2m ago 16m 73.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:18:03.511 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (16m) 2m ago 16m 72.8M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:18:03.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:03 smithi150 ceph-mon[25582]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:03.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:03 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:03.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:03 smithi139 ceph-mon[24727]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:03.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:03 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:03 smithi139 ceph-mon[29453]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:03 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:04.276 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T06:18:05.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:05 smithi139 ceph-mon[24727]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:18:05.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:05 smithi139 ceph-mon[29453]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:18:05.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:05 smithi150 ceph-mon[25582]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:18:06.332 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:06 smithi139 ceph-mon[24727]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:06.333 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:18:06.333 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:18:06.333 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T06:18:06.333 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:18:06.333 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:18:06.333 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:18:06.333 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:18:06.334 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:18:06.334 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:18:06.334 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:18:06.334 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": {}, 2024-02-10T06:18:06.334 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:18:06.334 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:18:06.334 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:18:06.334 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:18:06.334 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T06:18:06.335 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:18:06.335 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:18:06.335 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:18:06.335 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:06 smithi139 ceph-mon[29453]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:06.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:06 smithi150 ceph-mon[25582]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:06.941 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph -s' 2024-02-10T06:18:07.250 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:07 smithi139 ceph-mon[24727]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:07.250 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:07 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2348262168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:18:07.250 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:07 smithi139 ceph-mon[29453]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:07.250 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:07 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2348262168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:18:07.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:07 smithi150 ceph-mon[25582]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:07.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:07 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2348262168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:18:08.876 INFO:teuthology.orchestra.run.smithi139.stdout: cluster: 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: id: ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: health: HEALTH_OK 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: services: 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: mon: 3 daemons, quorum a,c,b (age 21m) 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: mgr: y(active, since 3m), standbys: x 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: osd: 8 osds: 8 up (since 18m), 8 in (since 18m) 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: data: 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: pools: 6 pools, 161 pgs 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: objects: 209 objects, 585 KiB 2024-02-10T06:18:08.877 INFO:teuthology.orchestra.run.smithi139.stdout: usage: 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:08.878 INFO:teuthology.orchestra.run.smithi139.stdout: pgs: 161 active+clean 2024-02-10T06:18:08.878 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:18:08.878 INFO:teuthology.orchestra.run.smithi139.stdout: io: 2024-02-10T06:18:08.878 INFO:teuthology.orchestra.run.smithi139.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-10T06:18:08.878 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-02-10T06:18:09.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:18:08] "GET /metrics HTTP/1.1" 200 37911 "" "Prometheus/2.33.4" 2024-02-10T06:18:09.414 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T06:18:09.440 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:09 smithi139 ceph-mon[24727]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:09.440 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:09 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1646429524' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:18:09.440 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:09 smithi139 ceph-mon[29453]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:09.440 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:09 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1646429524' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:18:09.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:09 smithi150 ceph-mon[25582]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:09.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:09 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1646429524' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:18:11.334 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_OK 2024-02-10T06:18:11.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:11 smithi139 ceph-mon[29453]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:11.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:11 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3626830584' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:18:11.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:11 smithi139 ceph-mon[24727]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:11.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:11 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3626830584' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:18:11.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:11 smithi150 ceph-mon[25582]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:11.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:11 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3626830584' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:18:11.926 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-02-10T06:18:13.525 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:13 smithi139 ceph-mon[24727]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:13.525 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:13 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:13.525 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:13 smithi139 ceph-mon[24727]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:13.525 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:13 smithi139 ceph-mon[29453]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:13.525 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:13 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:13.525 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:13 smithi139 ceph-mon[29453]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:13 smithi150 ceph-mon[25582]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:13 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:13 smithi150 ceph-mon[25582]: from='mgr.15201 172.21.15.139:0/3575844557' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:14.016 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:18:14.575 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:14 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3198527141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:18:14.575 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:14 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3198527141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:18:14.599 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph mgr fail' 2024-02-10T06:18:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:14 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3198527141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:18:16.282 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:16 smithi139 ceph-mon[24727]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:16.283 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:16 smithi139 ceph-mon[29453]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:16.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:16 smithi150 ceph-mon[25582]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:17.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:17 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3578101882' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:18:17.303 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:17 smithi139 ceph-mon[24727]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T06:18:17.303 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:17 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:17.060+0000 7ff3f39e8700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-10T06:18:17.303 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:17 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ignoring --setuser ceph since I am not root 2024-02-10T06:18:17.303 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:17 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ignoring --setgroup ceph since I am not root 2024-02-10T06:18:17.303 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:17 smithi139 ceph-mgr[63877]: -- 172.21.15.139:0/1464078429 <== mon.0 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fdcb63a1a0 con 0x55fdcb610800 2024-02-10T06:18:17.304 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:17 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3578101882' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:18:17.304 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:17 smithi139 ceph-mon[29453]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T06:18:17.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:17 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3578101882' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-10T06:18:17.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:17 smithi150 ceph-mon[25582]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T06:18:17.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:18:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:18:17] ENGINE Bus STOPPING 2024-02-10T06:18:17.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:18:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:18:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T06:18:17.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:18:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:18:17] ENGINE Bus STOPPED 2024-02-10T06:18:17.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:18:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:18:17] ENGINE Bus STARTING 2024-02-10T06:18:17.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:18:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:18:17] ENGINE Serving on http://:::9283 2024-02-10T06:18:17.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:18:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:18:17] ENGINE Bus STARTED 2024-02-10T06:18:17.705 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'sleep 180' 2024-02-10T06:18:18.031 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:17 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:17.659+0000 7fa00fe76200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T06:18:18.031 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:17 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:17.763+0000 7fa00fe76200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T06:18:18.290 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:18:18.291 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3578101882' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: mgrmap e33: x(active, starting, since 0.505359s) 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: Manager daemon x is now available 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:18:18.292 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:18:18.293 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3578101882' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: mgrmap e33: x(active, starting, since 0.505359s) 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: Manager daemon x is now available 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:18:18.294 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:18:18.332 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3578101882' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-10T06:18:18.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: mgrmap e33: x(active, starting, since 0.505359s) 2024-02-10T06:18:18.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:18:18.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:18:18.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:18:18.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: Manager daemon x is now available 2024-02-10T06:18:18.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:18.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:18.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:18.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:18:18.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:18:18.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:18:18.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:18 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T06:18:18.560 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:18.465+0000 7fa00fe76200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T06:18:18.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:18.559+0000 7fa00fe76200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T06:18:18.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:18.707+0000 7fa00fe76200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T06:18:18.837 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:18.806+0000 7fa00fe76200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T06:18:19.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:19 smithi150 ceph-mon[25582]: [10/Feb/2024:06:18:17] ENGINE Bus STARTING 2024-02-10T06:18:19.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:19 smithi150 ceph-mon[25582]: [10/Feb/2024:06:18:17] ENGINE Serving on https://172.21.15.150:7150 2024-02-10T06:18:19.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:19 smithi150 ceph-mon[25582]: [10/Feb/2024:06:18:17] ENGINE Serving on http://172.21.15.150:8765 2024-02-10T06:18:19.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:19 smithi150 ceph-mon[25582]: [10/Feb/2024:06:18:17] ENGINE Bus STARTED 2024-02-10T06:18:19.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:19 smithi150 ceph-mon[25582]: mgrmap e34: x(active, since 1.50587s) 2024-02-10T06:18:19.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:19 smithi150 ceph-mon[25582]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:19.169 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[24727]: [10/Feb/2024:06:18:17] ENGINE Bus STARTING 2024-02-10T06:18:19.169 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[24727]: [10/Feb/2024:06:18:17] ENGINE Serving on https://172.21.15.150:7150 2024-02-10T06:18:19.170 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[24727]: [10/Feb/2024:06:18:17] ENGINE Serving on http://172.21.15.150:8765 2024-02-10T06:18:19.170 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[24727]: [10/Feb/2024:06:18:17] ENGINE Bus STARTED 2024-02-10T06:18:19.170 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[24727]: mgrmap e34: x(active, since 1.50587s) 2024-02-10T06:18:19.170 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[24727]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:19.170 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:18.891+0000 7fa00fe76200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T06:18:19.170 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[29453]: [10/Feb/2024:06:18:17] ENGINE Bus STARTING 2024-02-10T06:18:19.170 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[29453]: [10/Feb/2024:06:18:17] ENGINE Serving on https://172.21.15.150:7150 2024-02-10T06:18:19.170 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[29453]: [10/Feb/2024:06:18:17] ENGINE Serving on http://172.21.15.150:8765 2024-02-10T06:18:19.170 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[29453]: [10/Feb/2024:06:18:17] ENGINE Bus STARTED 2024-02-10T06:18:19.171 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[29453]: mgrmap e34: x(active, since 1.50587s) 2024-02-10T06:18:19.171 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-mon[29453]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:19.479 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:19.176+0000 7fa00fe76200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T06:18:19.480 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:19.288+0000 7fa00fe76200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T06:18:19.480 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:19.380+0000 7fa00fe76200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T06:18:19.804 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:19.478+0000 7fa00fe76200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T06:18:19.804 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:19 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:19.659+0000 7fa00fe76200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T06:18:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:20 smithi139 ceph-mon[24727]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:20 smithi139 ceph-mon[24727]: mgrmap e35: x(active, since 2s) 2024-02-10T06:18:20.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:20 smithi139 ceph-mon[29453]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:20.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:20 smithi139 ceph-mon[29453]: mgrmap e35: x(active, since 2s) 2024-02-10T06:18:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:20 smithi150 ceph-mon[25582]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:20 smithi150 ceph-mon[25582]: mgrmap e35: x(active, since 2s) 2024-02-10T06:18:21.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:20.671+0000 7fa00fe76200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T06:18:21.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:20 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:20.806+0000 7fa00fe76200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T06:18:21.439 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:21 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:21.183+0000 7fa00fe76200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T06:18:21.439 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:21 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:21.276+0000 7fa00fe76200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T06:18:21.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:21 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:21.484+0000 7fa00fe76200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:18:21.837 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:21 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:21.678+0000 7fa00fe76200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T06:18:22.261 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:21 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:21.988+0000 7fa00fe76200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:18:22.261 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:22 smithi139 ceph-mon[24727]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:22.261 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:22 smithi139 ceph-mon[24727]: mgrmap e36: x(active, since 4s) 2024-02-10T06:18:22.262 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:22 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:22.262 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:22 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:22.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:22 smithi139 ceph-mon[29453]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:22.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:22 smithi139 ceph-mon[29453]: mgrmap e36: x(active, since 4s) 2024-02-10T06:18:22.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:22 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:22.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:22 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:22.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:22 smithi150 ceph-mon[25582]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:22.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:22 smithi150 ceph-mon[25582]: mgrmap e36: x(active, since 4s) 2024-02-10T06:18:22.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:22 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:22.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:22 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:22.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:22.260+0000 7fa00fe76200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T06:18:22.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:22.510+0000 7fa00fe76200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T06:18:23.683 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:23.371+0000 7fa00fe76200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T06:18:23.980 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[24727]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:23.981 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:23.981 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:23.981 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[24727]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:23.981 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:23.981 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:23.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[29453]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:23.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:23.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:23.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[29453]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:23.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:23.982 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:24.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:23 smithi150 ceph-mon[25582]: from='client.25183 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:23 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:23 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:23 smithi150 ceph-mon[25582]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:18:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:23 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:23 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:24.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:23.979+0000 7fa00fe76200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T06:18:24.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:24 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:24.085+0000 7fa00fe76200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T06:18:24.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:24 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:24.335+0000 7fa00fe76200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T06:18:24.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:24 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:18:24] ENGINE Bus STARTING 2024-02-10T06:18:24.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:24 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: CherryPy Checker: 2024-02-10T06:18:24.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:24 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: The Application mounted at '' has an empty config. 2024-02-10T06:18:24.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:24 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:18:24.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:24 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:18:24] ENGINE Serving on http://:::9283 2024-02-10T06:18:24.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:24 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:18:24] ENGINE Bus STARTED 2024-02-10T06:18:25.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[29453]: Standby manager daemon y started 2024-02-10T06:18:25.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T06:18:25.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[24727]: Standby manager daemon y started 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:18:25.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:25.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:25.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:25.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:25 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:18:25.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:25 smithi150 ceph-mon[25582]: Standby manager daemon y started 2024-02-10T06:18:25.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:25 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T06:18:25.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:25 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:18:25.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:25 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T06:18:25.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:25 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:18:25.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:25 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:25.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:25 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:25.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:25 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:25.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:25 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:18:26.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: mgrmap e37: x(active, since 8s), standbys: y 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: mgrmap e37: x(active, since 8s), standbys: y 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:18:26.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.303 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.303 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.303 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.303 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.303 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:18:26.303 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:18:26.303 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:26 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:26.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: mgrmap e37: x(active, since 8s), standbys: y 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:26.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:18:26.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:18:26.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:26 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:27.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:27 smithi139 ceph-mon[24727]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:18:27.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:27 smithi139 ceph-mon[24727]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:18:27.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:27 smithi139 ceph-mon[29453]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:18:27.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:27 smithi139 ceph-mon[29453]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:18:27.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:27 smithi150 ceph-mon[25582]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:18:27.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:27 smithi150 ceph-mon[25582]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:18:28.252 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:28 smithi139 ceph-mon[24727]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:18:28.252 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:28 smithi139 ceph-mon[29453]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:18:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:28 smithi150 ceph-mon[25582]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:18:28.917 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:18:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:18:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T06:18:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:30 smithi139 ceph-mon[29453]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:18:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:30 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:30 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:30.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:30 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/781433995' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:18:30.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:30 smithi139 ceph-mon[24727]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:18:30.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:30 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:30.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:30 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:30.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:30 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/781433995' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:18:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:30 smithi150 ceph-mon[25582]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:18:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:30 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:30 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:30 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/781433995' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:18:31.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:31 smithi150 ceph-mon[25582]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:18:31.055 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:31 smithi150 ceph-mon[25582]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:18:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:31 smithi139 ceph-mon[29453]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:18:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:31 smithi139 ceph-mon[29453]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:18:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:31 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/4287685213' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1533033186"}]: dispatch 2024-02-10T06:18:31.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:31 smithi139 ceph-mon[24727]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:18:31.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:31 smithi139 ceph-mon[24727]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:18:31.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:31 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/4287685213' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1533033186"}]: dispatch 2024-02-10T06:18:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:31 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/4287685213' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1533033186"}]: dispatch 2024-02-10T06:18:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:32 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/4287685213' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1533033186"}]': finished 2024-02-10T06:18:32.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:32 smithi139 ceph-mon[29453]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T06:18:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:32 smithi139 ceph-mon[29453]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-10T06:18:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:32 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1492381112' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2609372381"}]: dispatch 2024-02-10T06:18:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:32 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/4287685213' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1533033186"}]': finished 2024-02-10T06:18:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:32 smithi139 ceph-mon[24727]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T06:18:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:32 smithi139 ceph-mon[24727]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-10T06:18:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:32 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1492381112' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2609372381"}]: dispatch 2024-02-10T06:18:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:32 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/4287685213' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1533033186"}]': finished 2024-02-10T06:18:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:32 smithi150 ceph-mon[25582]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T06:18:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:32 smithi150 ceph-mon[25582]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-10T06:18:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:32 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1492381112' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2609372381"}]: dispatch 2024-02-10T06:18:33.209 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 systemd[1]: Stopping Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:18:33.209 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 bash[57694]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T06:18:33.209 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1492381112' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2609372381"}]': finished 2024-02-10T06:18:33.209 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-mon[25582]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T06:18:33.209 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:33.209 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/340612828' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2704800563"}]: dispatch 2024-02-10T06:18:33.209 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2704800563"}]: dispatch 2024-02-10T06:18:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:33 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1492381112' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2609372381"}]': finished 2024-02-10T06:18:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:33 smithi139 ceph-mon[29453]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T06:18:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:33 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:33.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:33 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/340612828' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2704800563"}]: dispatch 2024-02-10T06:18:33.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:33 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2704800563"}]: dispatch 2024-02-10T06:18:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:33 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1492381112' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2609372381"}]': finished 2024-02-10T06:18:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:33 smithi139 ceph-mon[24727]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T06:18:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:33 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:33 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/340612828' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2704800563"}]: dispatch 2024-02-10T06:18:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:33 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2704800563"}]: dispatch 2024-02-10T06:18:33.573 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.208Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:18:33.574 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.208Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T06:18:33.574 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.208Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T06:18:33.574 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.208Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T06:18:33.574 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.208Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T06:18:33.574 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.208Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T06:18:33.574 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.208Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T06:18:33.574 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.208Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T06:18:33.574 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.208Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T06:18:33.574 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.209Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T06:18:33.574 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.209Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T06:18:33.574 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[54753]: ts=2024-02-10T06:18:33.209Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T06:18:33.575 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 podman[57707]: 2024-02-10 06:18:33.218518316 +0000 UTC m=+0.122276313 container died 08b74b342924689fd7b68aa35d527add9bc6271fa2f14c5ce77c775621a0bdf2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:18:33.874 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 podman[57707]: 2024-02-10 06:18:33.599331181 +0000 UTC m=+0.503089187 container cleanup 08b74b342924689fd7b68aa35d527add9bc6271fa2f14c5ce77c775621a0bdf2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:18:33.874 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 bash[57707]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T06:18:33.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 bash[57738]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T06:18:33.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 podman[57721]: 2024-02-10 06:18:33.800414133 +0000 UTC m=+0.583784657 container remove 08b74b342924689fd7b68aa35d527add9bc6271fa2f14c5ce77c775621a0bdf2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:18:33.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-10T06:18:33.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 systemd[1]: Stopped Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:18:33.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Consumed 1.273s CPU time. 2024-02-10T06:18:33.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:33 smithi150 systemd[1]: Starting Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:18:34.278 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-mon[25582]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:18:34.279 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-mon[25582]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2704800563"}]': finished 2024-02-10T06:18:34.279 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-mon[25582]: osdmap e106: 8 total, 8 up, 8 in 2024-02-10T06:18:34.279 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2378308641' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/568329002"}]: dispatch 2024-02-10T06:18:34.279 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 podman[57805]: 2024-02-10 06:18:34.018151878 +0000 UTC m=+0.097438018 container create ec43e9d95e502e8f0185fea4d2dcdc85513b7d3023d8edc39719c60a90a221d0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:18:34.279 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 podman[57805]: 2024-02-10 06:18:33.939732387 +0000 UTC m=+0.019018500 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-10T06:18:34.279 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 podman[57805]: 2024-02-10 06:18:34.204017003 +0000 UTC m=+0.283303113 container init ec43e9d95e502e8f0185fea4d2dcdc85513b7d3023d8edc39719c60a90a221d0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:18:34.279 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 podman[57805]: 2024-02-10 06:18:34.206577724 +0000 UTC m=+0.285863828 container start ec43e9d95e502e8f0185fea4d2dcdc85513b7d3023d8edc39719c60a90a221d0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:18:34.279 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.230Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-10T06:18:34.279 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.230Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-10T06:18:34.279 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.230Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-10T06:18:34.280 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.230Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi150 (none))" 2024-02-10T06:18:34.280 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.230Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T06:18:34.280 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.230Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T06:18:34.280 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.232Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T06:18:34.280 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.232Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-10T06:18:34.280 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.234Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-10T06:18:34.280 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.235Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T06:18:34.280 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.235Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.49µs 2024-02-10T06:18:34.280 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.235Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T06:18:34.280 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 bash[57805]: ec43e9d95e502e8f0185fea4d2dcdc85513b7d3023d8edc39719c60a90a221d0 2024-02-10T06:18:34.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:34 smithi139 ceph-mon[24727]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:18:34.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:34 smithi139 ceph-mon[24727]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2704800563"}]': finished 2024-02-10T06:18:34.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:34 smithi139 ceph-mon[24727]: osdmap e106: 8 total, 8 up, 8 in 2024-02-10T06:18:34.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:34 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2378308641' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/568329002"}]: dispatch 2024-02-10T06:18:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:34 smithi139 ceph-mon[29453]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:18:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:34 smithi139 ceph-mon[29453]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/2704800563"}]': finished 2024-02-10T06:18:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:34 smithi139 ceph-mon[29453]: osdmap e106: 8 total, 8 up, 8 in 2024-02-10T06:18:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:34 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2378308641' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/568329002"}]: dispatch 2024-02-10T06:18:34.624 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 systemd[1]: Started Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:18:34.624 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.430Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-02-10T06:18:35.082 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.672Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-02-10T06:18:35.082 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.720Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-02-10T06:18:35.082 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.774Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-02-10T06:18:35.082 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.774Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-02-10T06:18:35.082 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.774Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=47.999µs wal_replay_duration=538.737759ms total_replay_duration=538.808481ms 2024-02-10T06:18:35.082 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.780Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T06:18:35.082 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.780Z caller=main.go:947 level=info msg="TSDB started" 2024-02-10T06:18:35.083 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.780Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T06:18:35.083 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.801Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.145578ms db_storage=969ns remote_storage=1.704µs web_handler=440ns query_engine=1.124µs scrape=378.497µs scrape_sd=124.868µs notify=18.005µs notify_sd=18.27µs rules=20.239127ms 2024-02-10T06:18:35.083 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:18:34 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:18:34.801Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-10T06:18:35.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2378308641' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/568329002"}]': finished 2024-02-10T06:18:35.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[24727]: osdmap e107: 8 total, 8 up, 8 in 2024-02-10T06:18:35.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:35.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:35.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:18:35.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:18:35.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:18:35.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:35.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2327860588' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3942310209"}]: dispatch 2024-02-10T06:18:35.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2378308641' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/568329002"}]': finished 2024-02-10T06:18:35.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[29453]: osdmap e107: 8 total, 8 up, 8 in 2024-02-10T06:18:35.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:35.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:35.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:18:35.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:18:35.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:18:35.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:35.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:35 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2327860588' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3942310209"}]: dispatch 2024-02-10T06:18:35.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:35 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2378308641' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/568329002"}]': finished 2024-02-10T06:18:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:35 smithi150 ceph-mon[25582]: osdmap e107: 8 total, 8 up, 8 in 2024-02-10T06:18:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:35 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:35 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:35 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:18:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:35 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:18:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:35 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:18:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:35 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:35 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2327860588' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3942310209"}]: dispatch 2024-02-10T06:18:36.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:18:36.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:18:36.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:18:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[29453]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:18:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2327860588' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3942310209"}]': finished 2024-02-10T06:18:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[29453]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T06:18:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/25866656' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2609372381"}]: dispatch 2024-02-10T06:18:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:18:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:18:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:18:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[24727]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:18:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2327860588' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3942310209"}]': finished 2024-02-10T06:18:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[24727]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T06:18:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:36 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/25866656' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2609372381"}]: dispatch 2024-02-10T06:18:36.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:36 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:18:36.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:36 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:18:36.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:36 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:18:36.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:36 smithi150 ceph-mon[25582]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:18:36.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:36 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2327860588' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3942310209"}]': finished 2024-02-10T06:18:36.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:36 smithi150 ceph-mon[25582]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T06:18:36.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:36 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/25866656' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2609372381"}]: dispatch 2024-02-10T06:18:37.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:37 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/25866656' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2609372381"}]': finished 2024-02-10T06:18:37.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:37 smithi139 ceph-mon[29453]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T06:18:37.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:37 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/25866656' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2609372381"}]': finished 2024-02-10T06:18:37.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:37 smithi139 ceph-mon[24727]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T06:18:37.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:37 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/25866656' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2609372381"}]': finished 2024-02-10T06:18:37.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:37 smithi150 ceph-mon[25582]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T06:18:38.387 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:38 smithi139 ceph-mon[24727]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:18:38.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:38 smithi139 ceph-mon[29453]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:18:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:38 smithi150 ceph-mon[25582]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:18:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:18:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[29453]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:18:39.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:39.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:18:39.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:39 smithi139 ceph-mon[24727]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:18:39.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:39 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:39 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:39 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:39 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:39 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:39.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:39 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:18:39.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:39 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:18:39.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:39 smithi150 ceph-mon[25582]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:18:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:41 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:41 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:41 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:41.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:18:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:18:41] "GET /metrics HTTP/1.1" 200 37910 "" "Prometheus/2.33.4" 2024-02-10T06:18:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:42 smithi139 ceph-mon[24727]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-02-10T06:18:42.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:42 smithi139 ceph-mon[29453]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-02-10T06:18:42.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:42 smithi150 ceph-mon[25582]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-02-10T06:18:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:44 smithi139 ceph-mon[24727]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:44.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:44 smithi139 ceph-mon[29453]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:44 smithi150 ceph-mon[25582]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:46.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:46 smithi150 ceph-mon[25582]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:18:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:46 smithi139 ceph-mon[24727]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:18:46.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:46 smithi139 ceph-mon[29453]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:18:48.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:48 smithi139 ceph-mon[24727]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-02-10T06:18:48.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:48 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:48 smithi139 ceph-mon[29453]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-02-10T06:18:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:48 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:48 smithi150 ceph-mon[25582]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-02-10T06:18:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:48 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:50 smithi139 ceph-mon[24727]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:50 smithi139 ceph-mon[29453]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:50 smithi150 ceph-mon[25582]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:51 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:51.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:51 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:51 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:51.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:18:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:18:51] "GET /metrics HTTP/1.1" 200 37918 "" "Prometheus/2.33.4" 2024-02-10T06:18:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:52 smithi139 ceph-mon[24727]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:52 smithi139 ceph-mon[29453]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:52 smithi150 ceph-mon[25582]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:54.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:54 smithi139 ceph-mon[24727]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:54.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:54 smithi139 ceph-mon[29453]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:54 smithi150 ceph-mon[25582]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:56 smithi139 ceph-mon[24727]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:56.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:56 smithi139 ceph-mon[29453]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:56 smithi150 ceph-mon[25582]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:58.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:18:58 smithi139 ceph-mon[24727]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:58.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:18:58 smithi139 ceph-mon[29453]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:18:58 smithi150 ceph-mon[25582]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:00 smithi139 ceph-mon[24727]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:00.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:00 smithi139 ceph-mon[29453]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:00 smithi150 ceph-mon[25582]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:01.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:01 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:01 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:01.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:01 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:01.608 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:19:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:19:01] "GET /metrics HTTP/1.1" 200 37918 "" "Prometheus/2.33.4" 2024-02-10T06:19:02.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:02 smithi139 ceph-mon[29453]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:02.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:02 smithi139 ceph-mon[24727]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:02 smithi150 ceph-mon[25582]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:03 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:03.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:03 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:03 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:04.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:04 smithi139 ceph-mon[24727]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:04.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:04 smithi139 ceph-mon[29453]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:04.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:04 smithi150 ceph-mon[25582]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:06 smithi139 ceph-mon[24727]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:06 smithi139 ceph-mon[29453]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:06 smithi150 ceph-mon[25582]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:08 smithi139 ceph-mon[24727]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:08 smithi139 ceph-mon[29453]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:08 smithi150 ceph-mon[25582]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:10 smithi139 ceph-mon[24727]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:10.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:10 smithi139 ceph-mon[29453]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:10 smithi150 ceph-mon[25582]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:11 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:11 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:11 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:11.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:19:11 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:19:11] "GET /metrics HTTP/1.1" 200 37911 "" "Prometheus/2.33.4" 2024-02-10T06:19:12.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:12 smithi139 ceph-mon[24727]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:12.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:12 smithi139 ceph-mon[29453]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:12 smithi150 ceph-mon[25582]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:14.450 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:14 smithi139 ceph-mon[24727]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:14.450 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:14 smithi139 ceph-mon[29453]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:14 smithi150 ceph-mon[25582]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:16 smithi139 ceph-mon[24727]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:16 smithi139 ceph-mon[29453]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:16 smithi150 ceph-mon[25582]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:18 smithi139 ceph-mon[24727]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:18.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:18 smithi139 ceph-mon[29453]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:18.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:18 smithi150 ceph-mon[25582]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:20 smithi139 ceph-mon[24727]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:20 smithi139 ceph-mon[29453]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:20 smithi150 ceph-mon[25582]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:21 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:21 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:21 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:21.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:19:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:19:21] "GET /metrics HTTP/1.1" 200 37915 "" "Prometheus/2.33.4" 2024-02-10T06:19:22.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:22 smithi139 ceph-mon[24727]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:22.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:22 smithi139 ceph-mon[29453]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:22 smithi150 ceph-mon[25582]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:24 smithi139 ceph-mon[24727]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:24 smithi139 ceph-mon[29453]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:24.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:24 smithi150 ceph-mon[25582]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:26 smithi139 ceph-mon[24727]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:26.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:26 smithi139 ceph-mon[29453]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:26.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:26 smithi150 ceph-mon[25582]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:28 smithi139 ceph-mon[24727]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:28.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:28 smithi139 ceph-mon[29453]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:28.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:28 smithi150 ceph-mon[25582]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:30.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:30 smithi139 ceph-mon[24727]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:30 smithi139 ceph-mon[29453]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:30 smithi150 ceph-mon[25582]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:31.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:31 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:31 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:31 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:31.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:19:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:19:31] "GET /metrics HTTP/1.1" 200 37915 "" "Prometheus/2.33.4" 2024-02-10T06:19:32.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:32 smithi139 ceph-mon[24727]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:32 smithi139 ceph-mon[29453]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:32 smithi150 ceph-mon[25582]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:33.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:33 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:33.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:33 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:33 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:34.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:34 smithi139 ceph-mon[24727]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:34 smithi139 ceph-mon[29453]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:34 smithi150 ceph-mon[25582]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:36.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:36 smithi139 ceph-mon[24727]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:36 smithi139 ceph-mon[29453]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:36.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:36 smithi150 ceph-mon[25582]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:38 smithi139 ceph-mon[24727]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:38 smithi139 ceph-mon[29453]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:38 smithi150 ceph-mon[25582]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:39.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:39 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:19:39.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:39 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:19:39.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:39 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:19:40.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:40 smithi139 ceph-mon[24727]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:40.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:40 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:19:40.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:40 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:19:40.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:40 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:19:40.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:40 smithi139 ceph-mon[29453]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:40.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:40 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:19:40.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:40 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:19:40.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:40 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:19:40.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:40 smithi150 ceph-mon[25582]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:40.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:40 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:19:40.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:40 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:19:40.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:40 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:19:41.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:41 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:41.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:41 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:41.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:41 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:41.625 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:19:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:19:41] "GET /metrics HTTP/1.1" 200 37917 "" "Prometheus/2.33.4" 2024-02-10T06:19:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:42 smithi139 ceph-mon[24727]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:42 smithi139 ceph-mon[29453]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:42.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:42 smithi150 ceph-mon[25582]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:44 smithi139 ceph-mon[24727]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:44 smithi139 ceph-mon[29453]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:44 smithi150 ceph-mon[25582]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:46 smithi139 ceph-mon[24727]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:46.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:46 smithi139 ceph-mon[29453]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:46 smithi150 ceph-mon[25582]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:48 smithi139 ceph-mon[24727]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:48 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:48 smithi139 ceph-mon[29453]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:48 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:48 smithi150 ceph-mon[25582]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:48 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:50 smithi139 ceph-mon[24727]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:50.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:50 smithi139 ceph-mon[29453]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:50 smithi150 ceph-mon[25582]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:51 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:51.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:51 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:51 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:51.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:19:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:19:51] "GET /metrics HTTP/1.1" 200 37913 "" "Prometheus/2.33.4" 2024-02-10T06:19:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:52 smithi139 ceph-mon[24727]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:52.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:52 smithi139 ceph-mon[29453]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:52 smithi150 ceph-mon[25582]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:54.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:54 smithi139 ceph-mon[24727]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:54.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:54 smithi139 ceph-mon[29453]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:54 smithi150 ceph-mon[25582]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:56.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:56 smithi139 ceph-mon[29453]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:56.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:56 smithi139 ceph-mon[24727]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:56 smithi150 ceph-mon[25582]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:58.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:19:58 smithi139 ceph-mon[24727]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:58.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:19:58 smithi139 ceph-mon[29453]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:19:58 smithi150 ceph-mon[25582]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:00 smithi139 ceph-mon[24727]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:00 smithi139 ceph-mon[24727]: overall HEALTH_OK 2024-02-10T06:20:00.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:00 smithi139 ceph-mon[29453]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:00.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:00 smithi139 ceph-mon[29453]: overall HEALTH_OK 2024-02-10T06:20:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:00 smithi150 ceph-mon[25582]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:00 smithi150 ceph-mon[25582]: overall HEALTH_OK 2024-02-10T06:20:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:01 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:01 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:01.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:01 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:01.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:20:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:20:01] "GET /metrics HTTP/1.1" 200 37913 "" "Prometheus/2.33.4" 2024-02-10T06:20:02.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:02 smithi139 ceph-mon[24727]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:02.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:02 smithi139 ceph-mon[29453]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:02 smithi150 ceph-mon[25582]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:03 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:03.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:03 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:03 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:04.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:04 smithi139 ceph-mon[24727]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:04.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:04 smithi139 ceph-mon[29453]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:04.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:04 smithi150 ceph-mon[25582]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:06 smithi139 ceph-mon[24727]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:06 smithi139 ceph-mon[29453]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:06 smithi150 ceph-mon[25582]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:08 smithi139 ceph-mon[29453]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:08 smithi139 ceph-mon[24727]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:08 smithi150 ceph-mon[25582]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:10 smithi139 ceph-mon[24727]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:10.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:10 smithi139 ceph-mon[29453]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:10 smithi150 ceph-mon[25582]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:11 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:11 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:11 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:11.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:20:11 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:20:11] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-02-10T06:20:12.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:12 smithi139 ceph-mon[24727]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:12.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:12 smithi139 ceph-mon[29453]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:12 smithi150 ceph-mon[25582]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:14.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:14 smithi139 ceph-mon[24727]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:14 smithi139 ceph-mon[29453]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:14 smithi150 ceph-mon[25582]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:16 smithi139 ceph-mon[24727]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:16 smithi139 ceph-mon[29453]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:16 smithi150 ceph-mon[25582]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:18.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:18 smithi139 ceph-mon[29453]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:18.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:18 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:18 smithi139 ceph-mon[24727]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:18 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:18 smithi150 ceph-mon[25582]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:18 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:20 smithi139 ceph-mon[24727]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:20 smithi139 ceph-mon[29453]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:20 smithi150 ceph-mon[25582]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:21 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:21.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:21 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:21 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:21.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:20:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:20:21] "GET /metrics HTTP/1.1" 200 37910 "" "Prometheus/2.33.4" 2024-02-10T06:20:22.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:22 smithi139 ceph-mon[24727]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:22.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:22 smithi139 ceph-mon[29453]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:22 smithi150 ceph-mon[25582]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:24.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:24 smithi139 ceph-mon[24727]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:24.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:24 smithi139 ceph-mon[29453]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:24.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:24 smithi150 ceph-mon[25582]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:26 smithi139 ceph-mon[24727]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:26.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:26 smithi139 ceph-mon[29453]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:26.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:26 smithi150 ceph-mon[25582]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:28 smithi139 ceph-mon[24727]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:28.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:28 smithi139 ceph-mon[29453]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:28.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:28 smithi150 ceph-mon[25582]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:30.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:30 smithi139 ceph-mon[29453]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:30.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:30 smithi139 ceph-mon[24727]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:30 smithi150 ceph-mon[25582]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:31.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:31 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:31.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:31 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:31.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:31 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:31.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:20:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:20:31] "GET /metrics HTTP/1.1" 200 37910 "" "Prometheus/2.33.4" 2024-02-10T06:20:32.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:32 smithi139 ceph-mon[24727]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:32 smithi139 ceph-mon[29453]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:32 smithi150 ceph-mon[25582]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:33 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:33.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:33 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:33.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:33 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:34 smithi150 ceph-mon[25582]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:34.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:34 smithi139 ceph-mon[24727]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:34.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:34 smithi139 ceph-mon[29453]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:36.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:36 smithi139 ceph-mon[24727]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:36 smithi139 ceph-mon[29453]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:36.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:36 smithi150 ceph-mon[25582]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:38 smithi139 ceph-mon[24727]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:38 smithi139 ceph-mon[29453]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:38 smithi150 ceph-mon[25582]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:40 smithi139 ceph-mon[24727]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:40 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:20:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:40 smithi139 ceph-mon[29453]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:40 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:20:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:40 smithi150 ceph-mon[25582]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:40 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:20:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:41 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:41 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:20:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:41 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:20:41.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:41 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:20:41.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:41 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:41.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:41 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:20:41.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:41 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:20:41.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:41 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:20:41.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:20:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:20:41] "GET /metrics HTTP/1.1" 200 37924 "" "Prometheus/2.33.4" 2024-02-10T06:20:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:41 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:41 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:20:41.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:41 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:20:41.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:41 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:20:42.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:42 smithi150 ceph-mon[25582]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:42 smithi139 ceph-mon[24727]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:42 smithi139 ceph-mon[29453]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:44 smithi139 ceph-mon[24727]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:44.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:44 smithi139 ceph-mon[29453]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:44 smithi150 ceph-mon[25582]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:46 smithi139 ceph-mon[24727]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:46.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:46 smithi139 ceph-mon[29453]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:46 smithi150 ceph-mon[25582]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:48 smithi139 ceph-mon[24727]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:48.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:48 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:48 smithi139 ceph-mon[29453]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:48 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:48 smithi150 ceph-mon[25582]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:48.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:48 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:50.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:50 smithi139 ceph-mon[29453]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:50.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:50 smithi139 ceph-mon[24727]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:50 smithi150 ceph-mon[25582]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:51 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:51.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:20:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:20:51] "GET /metrics HTTP/1.1" 200 37923 "" "Prometheus/2.33.4" 2024-02-10T06:20:51.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:51 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:51.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:51 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:52 smithi150 ceph-mon[25582]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:52.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:52 smithi139 ceph-mon[24727]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:52.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:52 smithi139 ceph-mon[29453]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:54.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:54 smithi139 ceph-mon[24727]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:54.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:54 smithi139 ceph-mon[29453]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:54 smithi150 ceph-mon[25582]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:56 smithi139 ceph-mon[24727]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:56.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:56 smithi139 ceph-mon[29453]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:56 smithi150 ceph-mon[25582]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:58.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:20:58 smithi139 ceph-mon[24727]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:58.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:20:58 smithi139 ceph-mon[29453]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:20:58 smithi150 ceph-mon[25582]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:00 smithi150 ceph-mon[25582]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:00.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:00 smithi139 ceph-mon[24727]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:00.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:00 smithi139 ceph-mon[29453]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:01.282 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:01 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:01.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:01 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:01.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:01 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:01.592 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:21:01] "GET /metrics HTTP/1.1" 200 37923 "" "Prometheus/2.33.4" 2024-02-10T06:21:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:02 smithi150 ceph-mon[25582]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:02.488 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:02 smithi139 ceph-mon[24727]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:02.488 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:02 smithi139 ceph-mon[29453]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:03 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:03.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:03 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:03.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:03 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:04.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:04 smithi150 ceph-mon[25582]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:04 smithi139 ceph-mon[24727]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:04.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:04 smithi139 ceph-mon[29453]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:06 smithi139 ceph-mon[24727]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:06 smithi139 ceph-mon[29453]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:06 smithi150 ceph-mon[25582]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:08 smithi139 ceph-mon[24727]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:08 smithi139 ceph-mon[29453]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:08 smithi150 ceph-mon[25582]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:10 smithi150 ceph-mon[25582]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:10.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:10 smithi139 ceph-mon[24727]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:10 smithi139 ceph-mon[29453]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:11 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:11.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:11 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:21:11] "GET /metrics HTTP/1.1" 200 37925 "" "Prometheus/2.33.4" 2024-02-10T06:21:11.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:11 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:11.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:11 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:12 smithi150 ceph-mon[25582]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:12.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:12 smithi139 ceph-mon[24727]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:12.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:12 smithi139 ceph-mon[29453]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:14.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:14 smithi139 ceph-mon[24727]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:14 smithi139 ceph-mon[29453]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:14 smithi150 ceph-mon[25582]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:16 smithi139 ceph-mon[24727]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:16 smithi139 ceph-mon[29453]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:16 smithi150 ceph-mon[25582]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:17.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:17 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:17.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:17 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:17.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:17 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:18 smithi139 ceph-mon[24727]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:18.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:18 smithi139 ceph-mon[29453]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:18.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:18 smithi150 ceph-mon[25582]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:20.190 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:21:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:20 smithi150 ceph-mon[25582]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:20.503 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:20 smithi139 ceph-mon[24727]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:20.503 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:20 smithi139 ceph-mon[29453]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:21.250 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:21 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:21.250 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:21 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:21.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:21 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:21.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:21:21] "GET /metrics HTTP/1.1" 200 37925 "" "Prometheus/2.33.4" 2024-02-10T06:21:22.240 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (12m) 2m ago 20m 22.3M - 0.23.0 ba2b418f427c 5d80a4f13ac5 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (12m) 2m ago 20m 53.1M - 8.3.5 dad864ee21e9 8b8f541bca1a 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (2m) 2m ago 19m 70.3M - 3.5 e1d6a67b021e e55ac7083290 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283 running (15m) 2m ago 24m 491M - 19.0.0-1361-g290212a1 c328ab9dcfc7 b3cac4d40883 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (9m) 2m ago 26m 447M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (26m) 2m ago 26m 72.5M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (25m) 2m ago 25m 52.0M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (25m) 2m ago 25m 52.1M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (12m) 2m ago 21m 17.1M - 1.3.1 1dbe0e931976 f2db304d3077 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (12m) 2m ago 21m 19.2M - 1.3.1 1dbe0e931976 0758bcdcf70a 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (24m) 2m ago 24m 53.5M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:21:22.241 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (24m) 2m ago 24m 53.8M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:21:22.242 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (23m) 2m ago 23m 52.7M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:21:22.242 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (23m) 2m ago 23m 52.0M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:21:22.242 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (23m) 2m ago 23m 53.3M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:21:22.242 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (22m) 2m ago 22m 54.2M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:21:22.242 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (22m) 2m ago 22m 51.2M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:21:22.242 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (21m) 2m ago 21m 54.8M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:21:22.242 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (2m) 2m ago 21m 44.7M - 2.33.4 514e6a882f6e ec43e9d95e50 2024-02-10T06:21:22.242 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (19m) 2m ago 19m 74.7M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:21:22.242 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (19m) 2m ago 19m 73.8M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:21:22.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:22 smithi150 ceph-mon[25582]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:22.496 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:22 smithi139 ceph-mon[24727]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:22.496 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:22 smithi139 ceph-mon[29453]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:22.887 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ls' 2024-02-10T06:21:23.200 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:23 smithi139 ceph-mon[24727]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:23.200 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:23 smithi139 ceph-mon[29453]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:23 smithi150 ceph-mon[25582]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:24.372 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:24 smithi139 ceph-mon[24727]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:24.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:24 smithi139 ceph-mon[29453]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:24.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:24 smithi150 ceph-mon[25582]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:24.923 INFO:teuthology.orchestra.run.smithi139.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-10T06:21:24.923 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager ?:9093,9094 1/1 2m ago 21m smithi139=a;count:1 2024-02-10T06:21:24.923 INFO:teuthology.orchestra.run.smithi139.stdout:grafana ?:3000 1/1 2m ago 21m smithi150=a;count:1 2024-02-10T06:21:24.923 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo ?:5000 1/1 2m ago 19m count:1 2024-02-10T06:21:24.923 INFO:teuthology.orchestra.run.smithi139.stdout:mgr 2/2 2m ago 24m smithi139=y;smithi150=x;count:2 2024-02-10T06:21:24.923 INFO:teuthology.orchestra.run.smithi139.stdout:mon 3/3 2m ago 25m smithi139:172.21.15.139=a;smithi139:[v2:172.21.15.139:3301,v1:172.21.15.139:6790]=c;smithi150:172.21.15.150=b;count:3 2024-02-10T06:21:24.923 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter ?:9100 2/2 2m ago 21m smithi139=a;smithi150=b;count:2 2024-02-10T06:21:24.923 INFO:teuthology.orchestra.run.smithi139.stdout:osd 8 2m ago - 2024-02-10T06:21:24.923 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus ?:9095 1/1 2m ago 21m smithi150=a;count:1 2024-02-10T06:21:24.923 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo ?:8000 2/2 2m ago 19m count:2 2024-02-10T06:21:25.529 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T06:21:26.495 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:26 smithi139 ceph-mon[24727]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:26.495 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:26 smithi139 ceph-mon[24727]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:26.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:26 smithi139 ceph-mon[29453]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:26.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:26 smithi139 ceph-mon[29453]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:26.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:26 smithi150 ceph-mon[25582]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:26.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:26 smithi150 ceph-mon[25582]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:27.546 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:21:27.546 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:21:27.546 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T06:21:27.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:21:27.546 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:21:27.546 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:21:27.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:21:27.546 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:21:27.546 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:21:27.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:21:27.546 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": {}, 2024-02-10T06:21:27.547 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:21:27.547 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:21:27.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:21:27.547 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:21:27.547 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T06:21:27.547 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:21:27.547 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:21:27.547 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:21:28.143 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-02-10T06:21:28.449 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:28 smithi139 ceph-mon[24727]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:28.449 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:28 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1452361965' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:21:28.449 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:28 smithi139 ceph-mon[29453]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:28.449 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:28 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1452361965' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:21:28.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:28 smithi150 ceph-mon[25582]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:28.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:28 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1452361965' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:21:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:30 smithi150 ceph-mon[25582]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:30.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:30 smithi139 ceph-mon[24727]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:30.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:30 smithi139 ceph-mon[29453]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:31.242 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:31 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:31.243 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:31 smithi139 ceph-mon[24727]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:31 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:31.243 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:31 smithi139 ceph-mon[29453]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:31.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:31 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:21:31] "GET /metrics HTTP/1.1" 200 37925 "" "Prometheus/2.33.4" 2024-02-10T06:21:31.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:31 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:31.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:31 smithi150 ceph-mon[25582]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:32.043 INFO:teuthology.orchestra.run.smithi139.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:21:32.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[24727]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:32.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:21:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:21:32.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:21:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[29453]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:21:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:32.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:32.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:21:32.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:32 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:21:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:32 smithi150 ceph-mon[25582]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:32 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:21:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:32 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:32 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:32 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:21:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:32 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:21:32.757 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T06:21:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:33 smithi139 ceph-mon[29453]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:21:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:33 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:33 smithi139 ceph-mon[29453]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:21:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:33 smithi139 ceph-mon[24727]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:21:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:33 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:33 smithi139 ceph-mon[24727]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:21:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:33 smithi150 ceph-mon[25582]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:21:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:33 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:33 smithi150 ceph-mon[25582]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:21:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:34 smithi150 ceph-mon[25582]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:34.405 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:34 smithi139 ceph-mon[24727]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:34.405 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:34 smithi139 ceph-mon[29453]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:36.155 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:21:36.466 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-mon[25582]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:36.466 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:21:36.466 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:36.466 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-mon[25582]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T06:21:36.466 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T06:21:36.466 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-mon[25582]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:21:36.466 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-mon[25582]: Standby manager daemon x started 2024-02-10T06:21:36.479 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[29453]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:36.479 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:21:36.479 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:36.479 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[29453]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T06:21:36.479 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T06:21:36.479 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[29453]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:21:36.479 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[29453]: Standby manager daemon x started 2024-02-10T06:21:36.480 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[24727]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:36.480 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' 2024-02-10T06:21:36.480 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:36.480 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[24727]: from='mgr.34994 172.21.15.150:0/2045506303' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T06:21:36.480 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T06:21:36.480 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[24727]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:21:36.480 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-mon[24727]: Standby manager daemon x started 2024-02-10T06:21:36.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:21:36] ENGINE Bus STOPPING 2024-02-10T06:21:36.837 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:21:36] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T06:21:36.837 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:21:36] ENGINE Bus STOPPED 2024-02-10T06:21:36.837 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:21:36] ENGINE Bus STARTING 2024-02-10T06:21:36.837 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:21:36] ENGINE Serving on http://:::9283 2024-02-10T06:21:36.837 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:36 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:21:36] ENGINE Bus STARTED 2024-02-10T06:21:36.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:36.465+0000 7f5f7222d700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-10T06:21:36.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ignoring --setuser ceph since I am not root 2024-02-10T06:21:36.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: ignoring --setgroup ceph since I am not root 2024-02-10T06:21:36.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-mgr[45803]: -- 172.21.15.150:0/2155206680 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x559ab39b7080 con 0x559ab3980800 2024-02-10T06:21:36.875 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:36 smithi150 ceph-mgr[45803]: -- 172.21.15.150:0/2155206680 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559ab39aa1a0 con 0x559ab3980800 2024-02-10T06:21:37.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:37.054+0000 7fa92c1a7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T06:21:37.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:37.151+0000 7fa92c1a7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T06:21:37.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:21:37.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:21:37.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T06:21:37.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T06:21:37.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: Failing over to other MGR 2024-02-10T06:21:37.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:37.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.34994 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-10T06:21:37.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: mgrmap e38: y(active, starting, since 1s), standbys: x 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:21:37.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:21:37.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:21:37.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:21:37.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: Manager daemon y is now available 2024-02-10T06:21:37.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:37.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:37.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:37.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:21:37.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:21:37.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:21:37.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: Failing over to other MGR 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.34994 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: mgrmap e38: y(active, starting, since 1s), standbys: x 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:21:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: Manager daemon y is now available 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:37.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:37.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:37.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:21:37.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:21:37.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:21:37.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:21:37.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:21:37.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:21:37.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T06:21:37.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: Failing over to other MGR 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.34994 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: mgrmap e38: y(active, starting, since 1s), standbys: x 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:21:37.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: Manager daemon y is now available 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:21:37.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:37 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:21:37.850 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:21:37.850 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (12m) 2m ago 20m 22.3M - 0.23.0 ba2b418f427c 5d80a4f13ac5 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (12m) 2m ago 20m 53.1M - 8.3.5 dad864ee21e9 8b8f541bca1a 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (3m) 2m ago 19m 70.3M - 3.5 e1d6a67b021e e55ac7083290 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283 running (15m) 2m ago 25m 491M - 19.0.0-1361-g290212a1 c328ab9dcfc7 b3cac4d40883 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (9m) 2m ago 26m 447M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (26m) 2m ago 27m 72.5M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (25m) 2m ago 25m 52.0M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (25m) 2m ago 25m 52.1M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (12m) 2m ago 21m 17.1M - 1.3.1 1dbe0e931976 f2db304d3077 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (12m) 2m ago 21m 19.2M - 1.3.1 1dbe0e931976 0758bcdcf70a 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (24m) 2m ago 24m 53.5M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (24m) 2m ago 24m 53.8M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (24m) 2m ago 24m 52.7M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (23m) 2m ago 23m 52.0M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:21:37.851 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (23m) 2m ago 23m 53.3M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:21:37.852 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (22m) 2m ago 22m 54.2M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:21:37.852 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (22m) 2m ago 22m 51.2M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:21:37.852 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (22m) 2m ago 22m 54.8M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:21:37.852 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (3m) 2m ago 21m 44.7M - 2.33.4 514e6a882f6e ec43e9d95e50 2024-02-10T06:21:37.852 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (20m) 2m ago 20m 74.7M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:21:37.852 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (19m) 2m ago 19m 73.8M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:21:38.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:37.889+0000 7fa92c1a7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T06:21:38.125 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:37.986+0000 7fa92c1a7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T06:21:38.259 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:21:38.259 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:21:38.259 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T06:21:38.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:21:38.259 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": {}, 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:21:38.260 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:21:38.470 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:38.130+0000 7fa92c1a7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T06:21:38.471 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:38.239+0000 7fa92c1a7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T06:21:38.471 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:38.336+0000 7fa92c1a7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T06:21:38.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-mon[25582]: [10/Feb/2024:06:21:37] ENGINE Bus STARTING 2024-02-10T06:21:38.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-mon[25582]: [10/Feb/2024:06:21:37] ENGINE Serving on http://172.21.15.139:8765 2024-02-10T06:21:38.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-mon[25582]: [10/Feb/2024:06:21:37] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T06:21:38.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-mon[25582]: [10/Feb/2024:06:21:37] ENGINE Bus STARTED 2024-02-10T06:21:38.471 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-mon[25582]: mgrmap e39: y(active, since 2s), standbys: x 2024-02-10T06:21:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[24727]: [10/Feb/2024:06:21:37] ENGINE Bus STARTING 2024-02-10T06:21:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[24727]: [10/Feb/2024:06:21:37] ENGINE Serving on http://172.21.15.139:8765 2024-02-10T06:21:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[24727]: [10/Feb/2024:06:21:37] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T06:21:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[24727]: [10/Feb/2024:06:21:37] ENGINE Bus STARTED 2024-02-10T06:21:38.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[24727]: mgrmap e39: y(active, since 2s), standbys: x 2024-02-10T06:21:38.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[24727]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:21:38.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[24727]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:38.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[24727]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:38.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3493891739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:21:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[29453]: [10/Feb/2024:06:21:37] ENGINE Bus STARTING 2024-02-10T06:21:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[29453]: [10/Feb/2024:06:21:37] ENGINE Serving on http://172.21.15.139:8765 2024-02-10T06:21:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[29453]: [10/Feb/2024:06:21:37] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T06:21:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[29453]: [10/Feb/2024:06:21:37] ENGINE Bus STARTED 2024-02-10T06:21:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[29453]: mgrmap e39: y(active, since 2s), standbys: x 2024-02-10T06:21:38.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[29453]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:21:38.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[29453]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:38.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[29453]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:38.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:38 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3493891739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:21:38.607 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:21:38.607 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:21:38.607 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:21:38.607 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-10T06:21:38.607 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [ 2024-02-10T06:21:38.607 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr" 2024-02-10T06:21:38.607 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-02-10T06:21:38.607 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "2/2 daemons upgraded", 2024-02-10T06:21:38.607 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "", 2024-02-10T06:21:38.607 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:21:38.607 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:21:38.722 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:38.614+0000 7fa92c1a7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T06:21:38.723 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-mon[25582]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:21:38.723 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-mon[25582]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:38.723 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-mon[25582]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:38.723 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3493891739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:21:39.112 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:38.726+0000 7fa92c1a7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T06:21:39.112 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:38.814+0000 7fa92c1a7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T06:21:39.112 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:38.911+0000 7fa92c1a7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T06:21:39.112 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:39.111+0000 7fa92c1a7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T06:21:40.107 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:40 smithi139 ceph-mon[24727]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:21:40.107 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:40 smithi139 ceph-mon[24727]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:40 smithi150 ceph-mon[25582]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:21:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:40 smithi150 ceph-mon[25582]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:40.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:40 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:40.184+0000 7fa92c1a7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T06:21:40.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:40 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:40.321+0000 7fa92c1a7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T06:21:40.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:40 smithi139 ceph-mon[29453]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:21:40.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:40 smithi139 ceph-mon[29453]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:21:40.970 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:40 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:40.674+0000 7fa92c1a7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T06:21:40.971 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:40 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:40.765+0000 7fa92c1a7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T06:21:40.971 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:40 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:40.970+0000 7fa92c1a7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:21:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:41 smithi139 ceph-mon[24727]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:41 smithi139 ceph-mon[24727]: mgrmap e40: y(active, since 4s), standbys: x 2024-02-10T06:21:41.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:41 smithi139 ceph-mon[29453]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:41.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:41 smithi139 ceph-mon[29453]: mgrmap e40: y(active, since 4s), standbys: x 2024-02-10T06:21:41.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:41.166+0000 7fa92c1a7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T06:21:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:41 smithi150 ceph-mon[25582]: from='client.25276 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:41.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:41 smithi150 ceph-mon[25582]: mgrmap e40: y(active, since 4s), standbys: x 2024-02-10T06:21:41.747 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:41.466+0000 7fa92c1a7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:21:42.003 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:41.746+0000 7fa92c1a7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T06:21:42.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:42 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:42.002+0000 7fa92c1a7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T06:21:42.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:42 smithi150 ceph-mon[25582]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:21:42.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:42 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:42.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:42 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:42.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:42 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:42.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:42 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:42 smithi139 ceph-mon[24727]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:21:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:42 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:42.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:42 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:42.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:42 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:42.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:42 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:42 smithi139 ceph-mon[29453]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:21:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:42 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:42 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:42 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:42 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:42 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:42.870+0000 7fa92c1a7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T06:21:43.595 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:43.490+0000 7fa92c1a7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T06:21:43.595 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:43.594+0000 7fa92c1a7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T06:21:43.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-02-10T06:21:43.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:43.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:21:43.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:21:43.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:21:43.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:21:43.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:21:43.597 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:21:43.597 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-mon[25582]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:21:43.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[24727]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-02-10T06:21:43.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:43.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:43.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:21:43.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:21:43.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:21:43.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:21:43.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:21:43.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:21:43.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:43 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:21:43.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:43.813+0000 7fa92c1a7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T06:21:43.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:21:43] ENGINE Bus STARTING 2024-02-10T06:21:44.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: CherryPy Checker: 2024-02-10T06:21:44.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: The Application mounted at '' has an empty config. 2024-02-10T06:21:44.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: 2024-02-10T06:21:44.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:21:43] ENGINE Serving on http://:::9283 2024-02-10T06:21:44.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[45799]: [10/Feb/2024:06:21:43] ENGINE Bus STARTED 2024-02-10T06:21:44.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: Standby manager daemon x restarted 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: Standby manager daemon x started 2024-02-10T06:21:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:21:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:21:44.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:44.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:21:44.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: Standby manager daemon x restarted 2024-02-10T06:21:44.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: Standby manager daemon x started 2024-02-10T06:21:44.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:21:44.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:21:44.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:21:44.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:44 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:21:44.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: Standby manager daemon x restarted 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: Standby manager daemon x started 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:21:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:44 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/1911944525' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:21:45.769 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:45 smithi139 ceph-mon[24727]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 0 B/s wr, 186 op/s 2024-02-10T06:21:45.769 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:45 smithi139 ceph-mon[24727]: mgrmap e41: y(active, since 9s), standbys: x 2024-02-10T06:21:45.769 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:45 smithi139 ceph-mon[29453]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 0 B/s wr, 186 op/s 2024-02-10T06:21:45.769 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:45 smithi139 ceph-mon[29453]: mgrmap e41: y(active, since 9s), standbys: x 2024-02-10T06:21:45.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:45 smithi150 ceph-mon[25582]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 0 B/s wr, 186 op/s 2024-02-10T06:21:45.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:45 smithi150 ceph-mon[25582]: mgrmap e41: y(active, since 9s), standbys: x 2024-02-10T06:21:47.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:21:46.828+0000 7f9f95f8b700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:21:47.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Creating ceph-iscsi config... 2024-02-10T06:21:47.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:21:47.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:21:47.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:21:47.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:21:47.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:21:47.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return _run_code(code, main_globals, None, 2024-02-10T06:21:47.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:21:47.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: exec(code, run_globals) 2024-02-10T06:21:47.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:21:47.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:21:47.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:21:47.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:21:47.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:21:47.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:21:47.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:21:47.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:21:47.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:21:47.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:21:47.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:21:47.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:21:47.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:21:47.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return future.result(timeout) 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return self.__get_result() 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: raise self._exception 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: config_blobs=daemon_spec.final_config, 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Creating ceph-iscsi config... 2024-02-10T06:21:47.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return _run_code(code, main_globals, None, 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: exec(code, run_globals) 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:21:47.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:21:47.092 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:21:47.092 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:21:47.092 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:21:47.092 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.092 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:46 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:47 smithi139 ceph-mon[29453]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 0 B/s wr, 145 op/s 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:47 smithi139 ceph-mon[29453]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:21:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:21:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: return future.result(timeout) 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.__get_result() 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: raise self._exception 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:21:47.839 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.840 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.840 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:21:47.840 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:21:47.840 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:21:47.840 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:21:47.840 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:21:47.840 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:21:47.840 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:21:47.840 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:21:47.840 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:21:47.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:47 smithi139 ceph-mon[24727]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 0 B/s wr, 145 op/s 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:47 smithi139 ceph-mon[24727]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:21:47.841 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:21:47.842 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: return future.result(timeout) 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.__get_result() 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: raise self._exception 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:21:47.843 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:21:47.844 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.844 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.844 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:21:47.844 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:21:47.844 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:21:47.844 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:21:47.844 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:21:47.844 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:21:47.844 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:21:47.844 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:21:47.844 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:47 smithi139 ceph-mon[29453]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:47 smithi139 ceph-mon[29453]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.845 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:47 smithi139 ceph-mon[24727]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:21:47.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:47 smithi139 ceph-mon[24727]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:21:47.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:47 smithi150 ceph-mon[25582]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 0 B/s wr, 145 op/s 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:47 smithi150 ceph-mon[25582]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:21:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:21:47.876 INFO:journalctl@ceph.mon.b.smithi150.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: return future.result(timeout) 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.__get_result() 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: raise self._exception 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:21:47.877 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:21:47.878 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:21:47.879 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:21:47.879 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:21:47.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:47 smithi150 ceph-mon[25582]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T06:21:47.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:47 smithi150 ceph-mon[25582]: Reconfiguring daemon prometheus.a on smithi150 2024-02-10T06:21:50.177 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-mon[25582]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:21:50.439 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 systemd[1]: Stopping Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:21:50.439 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 bash[60621]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T06:21:50.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:50 smithi139 ceph-mon[24727]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:21:50.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:50 smithi139 ceph-mon[29453]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:21:50.707 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.439Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:21:50.707 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.439Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T06:21:50.707 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.439Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T06:21:50.708 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.439Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T06:21:50.708 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.439Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T06:21:50.708 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.439Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T06:21:50.708 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.439Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T06:21:50.708 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.439Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T06:21:50.708 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.439Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T06:21:50.708 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.440Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T06:21:50.708 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.440Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T06:21:50.708 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[57819]: ts=2024-02-10T06:21:50.440Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T06:21:50.708 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 podman[60633]: 2024-02-10 06:21:50.44936948 +0000 UTC m=+0.111564828 container died ec43e9d95e502e8f0185fea4d2dcdc85513b7d3023d8edc39719c60a90a221d0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:21:51.048 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 podman[60633]: 2024-02-10 06:21:50.797396164 +0000 UTC m=+0.459591528 container cleanup ec43e9d95e502e8f0185fea4d2dcdc85513b7d3023d8edc39719c60a90a221d0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:21:51.048 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 bash[60633]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T06:21:51.048 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:50 smithi150 bash[60658]: Error: no container with name or ID "ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T06:21:51.307 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 podman[60645]: 2024-02-10 06:21:51.048320343 +0000 UTC m=+0.599737849 container remove ec43e9d95e502e8f0185fea4d2dcdc85513b7d3023d8edc39719c60a90a221d0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:21:51.307 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-10T06:21:51.307 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 systemd[1]: Stopped Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:21:51.308 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Consumed 1.306s CPU time. 2024-02-10T06:21:51.308 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 systemd[1]: Starting Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:21:51.308 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 podman[60725]: 2024-02-10 06:21:51.202619919 +0000 UTC m=+0.020684603 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-10T06:21:51.308 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 podman[60725]: 2024-02-10 06:21:51.307539901 +0000 UTC m=+0.125604584 container create 7d2fef3be707e3940dd0be8127b439d807d762826cab10a6420a5a9835a87a87 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:21:51.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 podman[60725]: 2024-02-10 06:21:51.593570191 +0000 UTC m=+0.411634888 container init 7d2fef3be707e3940dd0be8127b439d807d762826cab10a6420a5a9835a87a87 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:21:51.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 podman[60725]: 2024-02-10 06:21:51.597672636 +0000 UTC m=+0.415737327 container start 7d2fef3be707e3940dd0be8127b439d807d762826cab10a6420a5a9835a87a87 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:21:51.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.623Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-10T06:21:51.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.623Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-10T06:21:51.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.623Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-10T06:21:51.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.623Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi150 (none))" 2024-02-10T06:21:51.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.623Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T06:21:51.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.623Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T06:21:51.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.625Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T06:21:51.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.625Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-10T06:21:51.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.627Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-10T06:21:51.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.629Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T06:21:51.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.629Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=652.885µs 2024-02-10T06:21:51.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.629Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T06:21:51.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 bash[60725]: 7d2fef3be707e3940dd0be8127b439d807d762826cab10a6420a5a9835a87a87 2024-02-10T06:21:51.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 systemd[1]: Started Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:21:51.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:51.838Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-02-10T06:21:52.349 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-mon[25582]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 109 op/s 2024-02-10T06:21:52.350 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:52.350 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:52.350 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:52.350 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:21:52.350 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:52.350 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:52.107Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-02-10T06:21:52.350 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:52.165Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-02-10T06:21:52.350 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:52.219Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-02-10T06:21:52.350 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:52.225Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-02-10T06:21:52.351 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:52.226Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-02-10T06:21:52.351 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:52.226Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=46.097µs wal_replay_duration=596.195823ms total_replay_duration=596.91055ms 2024-02-10T06:21:52.351 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:52.230Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T06:21:52.351 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:52.230Z caller=main.go:947 level=info msg="TSDB started" 2024-02-10T06:21:52.351 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:52.230Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T06:21:52.351 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:52.248Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.300994ms db_storage=829ns remote_storage=1.238µs web_handler=457ns query_engine=829ns scrape=393.526µs scrape_sd=87.211µs notify=16.059µs notify_sd=16.157µs rules=17.543067ms 2024-02-10T06:21:52.351 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:21:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:21:52.248Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-10T06:21:52.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[24727]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 109 op/s 2024-02-10T06:21:52.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:52.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:52.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:52.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:21:52.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:52.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[29453]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 109 op/s 2024-02-10T06:21:52.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:52.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:52.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:52.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:21:52.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:52 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:53 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:21:53.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:53 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:21:53.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:53 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:21:54.337 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:54 smithi150 ceph-mon[25582]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 109 op/s 2024-02-10T06:21:54.337 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:54 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:21:54.337 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:54 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:21:54.337 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:54 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:21:54.337 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:54 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:54.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:54 smithi139 ceph-mon[24727]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 109 op/s 2024-02-10T06:21:54.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:54 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:21:54.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:54 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:21:54.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:54 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:21:54.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:54 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:54.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:54 smithi139 ceph-mon[29453]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 109 op/s 2024-02-10T06:21:54.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:54 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:21:54.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:54 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:21:54.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:54 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:21:54.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:54 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:55.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:55 smithi150 systemd[1]: Stopping Ceph mgr.x for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:21:55.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:55 smithi150 ceph-mon[25582]: Upgrade: Updating mgr.x 2024-02-10T06:21:55.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:55 smithi150 ceph-mon[25582]: Deploying daemon mgr.x on smithi150 2024-02-10T06:21:55.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:55 smithi139 ceph-mon[24727]: Upgrade: Updating mgr.x 2024-02-10T06:21:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:55 smithi139 ceph-mon[24727]: Deploying daemon mgr.x on smithi150 2024-02-10T06:21:55.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:55 smithi139 ceph-mon[29453]: Upgrade: Updating mgr.x 2024-02-10T06:21:55.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:55 smithi139 ceph-mon[29453]: Deploying daemon mgr.x on smithi150 2024-02-10T06:21:55.802 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:55 smithi150 podman[61078]: 2024-02-10 06:21:55.484247413 +0000 UTC m=+0.145358847 container died b3cac4d40883350d7ba84d57e21e8974e6a66997876626cc457f3c9cb1e1c91b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-10T06:21:56.111 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:55 smithi150 podman[61078]: 2024-02-10 06:21:55.802899676 +0000 UTC m=+0.464011110 container cleanup b3cac4d40883350d7ba84d57e21e8974e6a66997876626cc457f3c9cb1e1c91b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-10T06:21:56.112 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:55 smithi150 bash[61078]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x 2024-02-10T06:21:56.112 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:55 smithi150 podman[61092]: 2024-02-10 06:21:55.987888306 +0000 UTC m=+0.505848619 container remove b3cac4d40883350d7ba84d57e21e8974e6a66997876626cc457f3c9cb1e1c91b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-10T06:21:56.112 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:55 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:21:56.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:56 smithi150 ceph-mon[25582]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:21:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:56 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-10T06:21:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:56 smithi150 systemd[1]: Stopped Ceph mgr.x for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:21:56.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:56 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.x.service: Consumed 57.247s CPU time. 2024-02-10T06:21:56.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:56 smithi139 ceph-mon[24727]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:21:56.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:56 smithi139 ceph-mon[29453]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:21:56.776 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:56 smithi150 systemd[1]: Starting Ceph mgr.x for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:21:56.776 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:56 smithi150 podman[61223]: 2024-02-10 06:21:56.553653683 +0000 UTC m=+0.018676358 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:21:56.776 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:56 smithi150 podman[61223]: 2024-02-10 06:21:56.655619567 +0000 UTC m=+0.120642244 container create f312c83e6dd43d1da1111ae653c6fcab28bfd4ab0b594320bba90f1d17c1112c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-10T06:21:57.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:56 smithi150 podman[61223]: 2024-02-10 06:21:56.883285136 +0000 UTC m=+0.348307810 container init f312c83e6dd43d1da1111ae653c6fcab28bfd4ab0b594320bba90f1d17c1112c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-10T06:21:57.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:56 smithi150 podman[61223]: 2024-02-10 06:21:56.886031316 +0000 UTC m=+0.351053997 container start f312c83e6dd43d1da1111ae653c6fcab28bfd4ab0b594320bba90f1d17c1112c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-10T06:21:57.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:56 smithi150 ceph-mgr[61242]: -- 172.21.15.150:0/3734075448 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562fae0341a0 con 0x562fae00b000 2024-02-10T06:21:57.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:56 smithi150 bash[61223]: f312c83e6dd43d1da1111ae653c6fcab28bfd4ab0b594320bba90f1d17c1112c 2024-02-10T06:21:57.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:56 smithi150 systemd[1]: Started Ceph mgr.x for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:21:57.848 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:57.454+0000 7f18583ed200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T06:21:57.848 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:57.559+0000 7f18583ed200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T06:21:58.110 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:58 smithi150 ceph-mon[25582]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:21:58.111 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:58 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:58.111 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:58 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:58.111 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:58 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:58 smithi139 ceph-mon[24727]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:21:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:58 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:58 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:58 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:58 smithi139 ceph-mon[29453]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:21:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:58 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:58 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:21:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:58 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:58.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:58.241+0000 7f18583ed200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T06:21:58.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:58.333+0000 7f18583ed200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T06:21:58.748 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:58.482+0000 7f18583ed200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T06:21:58.748 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:58.589+0000 7f18583ed200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T06:21:58.748 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:58.673+0000 7f18583ed200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T06:21:58.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:21:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:21:58] "GET /metrics HTTP/1.1" 200 37923 "" "Prometheus/2.33.4" 2024-02-10T06:21:59.032 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:58.952+0000 7f18583ed200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T06:21:59.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:21:59 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1904491882' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:21:59.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:21:59 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1904491882' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:21:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:21:59 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1904491882' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:21:59.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:59.063+0000 7f18583ed200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T06:21:59.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:59.152+0000 7f18583ed200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T06:21:59.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:59.267+0000 7f18583ed200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T06:21:59.777 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:21:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:21:59.470+0000 7f18583ed200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T06:22:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:00 smithi150 ceph-mon[25582]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:22:00.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:00 smithi139 ceph-mon[24727]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:22:00.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:00 smithi139 ceph-mon[29453]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:22:00.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:00.487+0000 7f18583ed200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T06:22:00.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:00.620+0000 7f18583ed200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T06:22:01.221 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:00.969+0000 7f18583ed200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T06:22:01.221 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:01.061+0000 7f18583ed200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T06:22:01.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:01.269+0000 7f18583ed200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:22:01.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:01.460+0000 7f18583ed200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T06:22:02.059 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:01.778+0000 7f18583ed200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:22:02.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:02 smithi150 ceph-mon[25582]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:22:02.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:02 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:02.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:02 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:02.316 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:02.057+0000 7f18583ed200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T06:22:02.316 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:02.313+0000 7f18583ed200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T06:22:02.389 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:02 smithi139 ceph-mon[24727]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:22:02.389 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:02 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:02.389 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:02 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:02.389 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:02 smithi139 ceph-mon[29453]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:22:02.389 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:02 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:02.389 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:02 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:03.539 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:03 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:03.182+0000 7f18583ed200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T06:22:03.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:03 smithi150 ceph-mon[25582]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:22:03.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:03 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:03.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:03 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:04.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:03 smithi139 ceph-mon[24727]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:22:04.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:03 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:04.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:03 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:04.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:03 smithi139 ceph-mon[29453]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:22:04.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:03 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:04.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:03 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:04.123 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:03 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:03.791+0000 7f18583ed200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T06:22:04.123 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:03 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:03.893+0000 7f18583ed200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T06:22:04.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:04.122+0000 7f18583ed200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T06:22:04.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: [10/Feb/2024:06:22:04] ENGINE Bus STARTING 2024-02-10T06:22:04.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: CherryPy Checker: 2024-02-10T06:22:04.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: The Application mounted at '' has an empty config. 2024-02-10T06:22:04.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:22:04.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: [10/Feb/2024:06:22:04] ENGINE Serving on http://:::9283 2024-02-10T06:22:04.375 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:22:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: [10/Feb/2024:06:22:04] ENGINE Bus STARTED 2024-02-10T06:22:05.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:05 smithi150 ceph-mon[25582]: Standby manager daemon x restarted 2024-02-10T06:22:05.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:05 smithi150 ceph-mon[25582]: Standby manager daemon x started 2024-02-10T06:22:05.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:05 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:22:05.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:05 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:22:05.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:05 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:22:05.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:05 smithi150 ceph-mon[25582]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:22:05.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[24727]: Standby manager daemon x restarted 2024-02-10T06:22:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[24727]: Standby manager daemon x started 2024-02-10T06:22:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:22:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:22:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:22:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[24727]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:22:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[29453]: Standby manager daemon x restarted 2024-02-10T06:22:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[29453]: Standby manager daemon x started 2024-02-10T06:22:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:22:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:22:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:22:05.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:05 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3645065284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:22:06.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:06 smithi150 ceph-mon[25582]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:22:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:06 smithi150 ceph-mon[25582]: mgrmap e42: y(active, since 29s), standbys: x 2024-02-10T06:22:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:06 smithi139 ceph-mon[24727]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:22:06.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:06 smithi139 ceph-mon[24727]: mgrmap e42: y(active, since 29s), standbys: x 2024-02-10T06:22:06.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:06 smithi139 ceph-mon[29453]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:22:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:06 smithi139 ceph-mon[29453]: mgrmap e42: y(active, since 29s), standbys: x 2024-02-10T06:22:07.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:07 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:07 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:07 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:07 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:07 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:07 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:07 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:07 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:07.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:07 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:07.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:07.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:07.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:07.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:07.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:07.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:07.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:07.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:07 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:08.332 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:08 smithi139 ceph-mon[24727]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:22:08.332 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:08 smithi139 ceph-mon[24727]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:22:08.332 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:08 smithi139 ceph-mon[24727]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:22:08.333 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:08 smithi139 ceph-mon[29453]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:22:08.333 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:08 smithi139 ceph-mon[29453]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:22:08.333 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:08 smithi139 ceph-mon[29453]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:22:08.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:08 smithi150 ceph-mon[25582]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:22:08.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:08 smithi150 ceph-mon[25582]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:22:08.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:08 smithi150 ceph-mon[25582]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-10T06:22:08.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:22:08] "GET /metrics HTTP/1.1" 200 37925 "" "Prometheus/2.33.4" 2024-02-10T06:22:08.967 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:22:09.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:22:09.384+0000 7f9f95f8b700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:09.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Creating ceph-iscsi config... 2024-02-10T06:22:09.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:09.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:09.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:09.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:09.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:09.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:22:09.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:09.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return _run_code(code, main_globals, None, 2024-02-10T06:22:09.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:09.588 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: exec(code, run_globals) 2024-02-10T06:22:09.588 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:09.588 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:09.588 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:09.588 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:09.588 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:09.588 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:09.588 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:09.588 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return future.result(timeout) 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return self.__get_result() 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:22:09.589 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: raise self._exception 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: config_blobs=daemon_spec.final_config, 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Creating ceph-iscsi config... 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return _run_code(code, main_globals, None, 2024-02-10T06:22:09.590 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:09.591 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: exec(code, run_globals) 2024-02-10T06:22:09.591 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:09.591 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:09.591 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:09.591 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:09.591 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:09.591 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:09.591 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:09.591 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:09.591 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:09.642 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:22:09.642 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (13m) 28s ago 21m 21.8M - 0.23.0 ba2b418f427c 5d80a4f13ac5 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (12m) 8s ago 21m 54.8M - 8.3.5 dad864ee21e9 8b8f541bca1a 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (3m) 28s ago 20m 49.2M - 3.5 e1d6a67b021e e55ac7083290 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (13s) 8s ago 25m 63.5M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (10m) 28s ago 27m 476M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (27m) 28s ago 27m 73.6M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (25m) 8s ago 25m 57.5M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (26m) 28s ago 26m 52.3M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (13m) 28s ago 21m 17.2M - 1.3.1 1dbe0e931976 f2db304d3077 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (13m) 8s ago 21m 19.1M - 1.3.1 1dbe0e931976 0758bcdcf70a 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (25m) 28s ago 25m 53.7M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (24m) 28s ago 24m 54.1M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (24m) 28s ago 24m 51.9M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (24m) 28s ago 24m 52.3M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (23m) 8s ago 23m 55.6M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:22:09.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (23m) 8s ago 23m 55.2M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:22:09.644 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (23m) 8s ago 23m 52.5M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:22:09.644 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (22m) 8s ago 22m 56.2M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:22:09.644 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (18s) 8s ago 22m 48.7M - 2.33.4 514e6a882f6e 7d2fef3be707 2024-02-10T06:22:09.644 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (20m) 28s ago 20m 75.4M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:22:09.644 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (20m) 8s ago 20m 74.9M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:22:10.055 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:22:10.055 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:22:10.055 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T06:22:10.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:22:10.055 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:22:10.055 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": {}, 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:22:10.056 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:22:10.333 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:10.333 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:10.333 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:10.333 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.333 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.334 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/304355563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T06:22:10.335 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:10 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/304355563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.393 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:22:10.393 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:22:10.393 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:22:10.393 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-10T06:22:10.393 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [ 2024-02-10T06:22:10.393 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr" 2024-02-10T06:22:10.393 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-02-10T06:22:10.394 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "2/2 daemons upgraded", 2024-02-10T06:22:10.394 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading node-exporter daemons", 2024-02-10T06:22:10.394 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:22:10.394 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:22:10.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:10.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:10.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:10.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T06:22:10.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T06:22:10.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-10T06:22:10.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:10.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:10.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:10 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/304355563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[24727]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[24727]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:22:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:22:11.338 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: return future.result(timeout) 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.__get_result() 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: raise self._exception 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:11.339 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:11.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:11.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:11.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:11.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:11.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:22:11.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:11.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:11.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:11.340 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[29453]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[29453]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:11.341 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:22:11.342 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: return future.result(timeout) 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.__get_result() 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: raise self._exception 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:11.343 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.344 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all mgr 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all crash 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all mds 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all nfs 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[29453]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.345 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all mgr 2024-02-10T06:22:11.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all crash 2024-02-10T06:22:11.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all mds 2024-02-10T06:22:11.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:22:11.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:22:11.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:22:11.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all nfs 2024-02-10T06:22:11.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:22:11.346 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:11 smithi139 ceph-mon[24727]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:11 smithi150 ceph-mon[25582]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:11 smithi150 ceph-mon[25582]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:11.625 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:22:11.626 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: return future.result(timeout) 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.__get_result() 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: raise self._exception 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:11.627 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:11.628 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:11.629 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:11.629 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:11.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:11 smithi150 ceph-mon[25582]: Upgrade: Setting container_image for all mgr 2024-02-10T06:22:11.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:11 smithi150 ceph-mon[25582]: Upgrade: Setting container_image for all crash 2024-02-10T06:22:11.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:11 smithi150 ceph-mon[25582]: Upgrade: Setting container_image for all mds 2024-02-10T06:22:11.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:11 smithi150 ceph-mon[25582]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:22:11.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:11 smithi150 ceph-mon[25582]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:22:11.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:11 smithi150 ceph-mon[25582]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:22:11.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:11 smithi150 ceph-mon[25582]: Upgrade: Setting container_image for all nfs 2024-02-10T06:22:11.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:11 smithi150 ceph-mon[25582]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:22:11.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:11 smithi150 ceph-mon[25582]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:11.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:11 smithi139 systemd[1]: Stopping Ceph node-exporter.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:22:12.308 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:12 smithi139 ceph-mon[24727]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:12.309 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:12 smithi139 ceph-mon[24727]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:12.309 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:12 smithi139 ceph-mon[24727]: Upgrade: Updating node-exporter.a (1/2) 2024-02-10T06:22:12.309 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:12 smithi139 ceph-mon[24727]: Deploying daemon node-exporter.a on smithi139 2024-02-10T06:22:12.309 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:12 smithi139 ceph-mon[29453]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:12.309 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:12 smithi139 ceph-mon[29453]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:12.309 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:12 smithi139 ceph-mon[29453]: Upgrade: Updating node-exporter.a (1/2) 2024-02-10T06:22:12.309 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:12 smithi139 ceph-mon[29453]: Deploying daemon node-exporter.a on smithi139 2024-02-10T06:22:12.310 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:12 smithi139 podman[77157]: 2024-02-10 06:22:12.003131871 +0000 UTC m=+0.134519699 container died f2db304d3077a5819956062b83fda3d3979d9cc9e64577114eef00eaee8190d4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:12.310 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:12 smithi139 podman[77157]: 2024-02-10 06:22:12.308751605 +0000 UTC m=+0.440139478 container cleanup f2db304d3077a5819956062b83fda3d3979d9cc9e64577114eef00eaee8190d4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:12.586 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:12 smithi139 bash[77157]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a 2024-02-10T06:22:12.586 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:12 smithi139 podman[77171]: 2024-02-10 06:22:12.526607512 +0000 UTC m=+0.523394696 container remove f2db304d3077a5819956062b83fda3d3979d9cc9e64577114eef00eaee8190d4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:12.586 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:12 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:22:12.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:12 smithi150 ceph-mon[25582]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:12.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:12 smithi150 ceph-mon[25582]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:12.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:12 smithi150 ceph-mon[25582]: Upgrade: Updating node-exporter.a (1/2) 2024-02-10T06:22:12.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:12 smithi150 ceph-mon[25582]: Deploying daemon node-exporter.a on smithi139 2024-02-10T06:22:13.086 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:12 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T06:22:13.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:12 smithi139 systemd[1]: Stopped Ceph node-exporter.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:22:13.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:12 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Consumed 5.232s CPU time. 2024-02-10T06:22:13.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:12 smithi139 systemd[1]: Starting Ceph node-exporter.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:22:13.485 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:13 smithi139 bash[77299]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-10T06:22:13.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:13 smithi139 bash[77299]: Getting image source signatures 2024-02-10T06:22:13.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:13 smithi139 bash[77299]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-10T06:22:13.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:13 smithi139 bash[77299]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-10T06:22:13.836 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:13 smithi139 bash[77299]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-10T06:22:14.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:14 smithi139 ceph-mon[24727]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-02-10T06:22:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:14 smithi139 ceph-mon[29453]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-02-10T06:22:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:14 smithi150 ceph-mon[25582]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-02-10T06:22:15.336 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:14 smithi139 bash[77299]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-10T06:22:15.336 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 bash[77299]: Writing manifest to image destination 2024-02-10T06:22:15.630 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 podman[77299]: 2024-02-10 06:22:15.3297279 +0000 UTC m=+2.304570018 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-10T06:22:15.630 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 podman[77299]: 2024-02-10 06:22:15.411653095 +0000 UTC m=+2.386495199 container create 3c4db3eedf5e9da4475f81aa92423573d6f198209060ff58cc0248b1fbfa9640 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:15.631 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 podman[77299]: 2024-02-10 06:22:15.630278299 +0000 UTC m=+2.605120413 container init 3c4db3eedf5e9da4475f81aa92423573d6f198209060ff58cc0248b1fbfa9640 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 podman[77299]: 2024-02-10 06:22:15.633563683 +0000 UTC m=+2.608405796 container start 3c4db3eedf5e9da4475f81aa92423573d6f198209060ff58cc0248b1fbfa9640 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.637Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.637Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=arp 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-10T06:22:16.087 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-10T06:22:16.088 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-10T06:22:16.088 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-10T06:22:16.088 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-10T06:22:16.088 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-10T06:22:16.088 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-10T06:22:16.088 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=edac 2024-02-10T06:22:16.088 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.638Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-10T06:22:16.089 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=os 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=stat 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=time 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=uname 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-10T06:22:16.090 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-10T06:22:16.091 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-10T06:22:16.091 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-10T06:22:16.091 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a[77349]: ts=2024-02-10T06:22:15.639Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-10T06:22:16.091 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 bash[77299]: 3c4db3eedf5e9da4475f81aa92423573d6f198209060ff58cc0248b1fbfa9640 2024-02-10T06:22:16.091 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:22:15 smithi139 systemd[1]: Started Ceph node-exporter.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:22:16.468 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:16 smithi150 ceph-mon[25582]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:22:16.468 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:16 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:16.468 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:16 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:16 smithi139 ceph-mon[24727]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:22:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:16 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:16 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:16 smithi139 ceph-mon[29453]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:22:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:16 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:16 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:18.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:18 smithi150 ceph-mon[25582]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:22:18.372 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:18 smithi150 systemd[1]: Stopping Ceph node-exporter.b for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:22:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:18 smithi139 ceph-mon[24727]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:22:18.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:18 smithi139 ceph-mon[29453]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:22:18.624 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:18 smithi150 podman[62618]: 2024-02-10 06:22:18.371972345 +0000 UTC m=+0.100498269 container died 0758bcdcf70aa02fcc554c33740fc7eb07f9173a58ca18c2bcaf7bc5302e7431 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:22:19.063 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:18 smithi150 podman[62618]: 2024-02-10 06:22:18.713286185 +0000 UTC m=+0.441812102 container cleanup 0758bcdcf70aa02fcc554c33740fc7eb07f9173a58ca18c2bcaf7bc5302e7431 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:22:19.063 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:18 smithi150 bash[62618]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b 2024-02-10T06:22:19.063 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:18 smithi150 podman[62631]: 2024-02-10 06:22:18.881027054 +0000 UTC m=+0.510267828 container remove 0758bcdcf70aa02fcc554c33740fc7eb07f9173a58ca18c2bcaf7bc5302e7431 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:22:19.063 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:18 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:22:19.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:22:18] "GET /metrics HTTP/1.1" 200 37925 "" "Prometheus/2.33.4" 2024-02-10T06:22:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:19 smithi150 ceph-mon[25582]: Upgrade: Updating node-exporter.b (2/2) 2024-02-10T06:22:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:19 smithi150 ceph-mon[25582]: Deploying daemon node-exporter.b on smithi150 2024-02-10T06:22:19.375 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:19 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T06:22:19.375 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:19 smithi150 systemd[1]: Stopped Ceph node-exporter.b for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:22:19.375 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:19 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Consumed 5.008s CPU time. 2024-02-10T06:22:19.375 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:19 smithi150 systemd[1]: Starting Ceph node-exporter.b for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:22:19.585 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:19 smithi139 ceph-mon[24727]: Upgrade: Updating node-exporter.b (2/2) 2024-02-10T06:22:19.585 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:19 smithi139 ceph-mon[24727]: Deploying daemon node-exporter.b on smithi150 2024-02-10T06:22:19.585 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:19 smithi139 ceph-mon[29453]: Upgrade: Updating node-exporter.b (2/2) 2024-02-10T06:22:19.585 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:19 smithi139 ceph-mon[29453]: Deploying daemon node-exporter.b on smithi150 2024-02-10T06:22:19.838 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:19 smithi150 bash[62759]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-10T06:22:20.124 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:19 smithi150 bash[62759]: Getting image source signatures 2024-02-10T06:22:20.124 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:19 smithi150 bash[62759]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-10T06:22:20.124 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:19 smithi150 bash[62759]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-10T06:22:20.124 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:19 smithi150 bash[62759]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-10T06:22:20.515 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:20 smithi139 ceph-mon[24727]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:22:20.516 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:20 smithi139 ceph-mon[29453]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:22:20.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:20 smithi150 ceph-mon[25582]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:22:21.395 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:21 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1310868490' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:22:21.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:21 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1310868490' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:22:21.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:21 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1310868490' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:22:21.874 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:21 smithi150 bash[62759]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-10T06:22:21.874 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:21 smithi150 bash[62759]: Writing manifest to image destination 2024-02-10T06:22:22.194 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-mon[25582]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T06:22:22.194 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:22.195 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:21 smithi150 podman[62759]: 2024-02-10 06:22:21.868154555 +0000 UTC m=+2.472676512 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-10T06:22:22.195 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:21 smithi150 podman[62759]: 2024-02-10 06:22:21.966866068 +0000 UTC m=+2.571388025 container create b250d2d9c43c27d6f36fb735b61baee03501f9569f62564506e3e0f136ebdac4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:22:22.453 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 podman[62759]: 2024-02-10 06:22:22.19407192 +0000 UTC m=+2.798593877 container init b250d2d9c43c27d6f36fb735b61baee03501f9569f62564506e3e0f136ebdac4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:22:22.453 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 podman[62759]: 2024-02-10 06:22:22.197104334 +0000 UTC m=+2.801626300 container start b250d2d9c43c27d6f36fb735b61baee03501f9569f62564506e3e0f136ebdac4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:22:22.453 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.201Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-10T06:22:22.453 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.201Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-10T06:22:22.453 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.201Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-10T06:22:22.453 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.201Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-10T06:22:22.454 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-10T06:22:22.454 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T06:22:22.454 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-10T06:22:22.454 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=arp 2024-02-10T06:22:22.454 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-10T06:22:22.454 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-10T06:22:22.454 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-10T06:22:22.454 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-10T06:22:22.454 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-10T06:22:22.454 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=edac 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-10T06:22:22.455 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=os 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-10T06:22:22.456 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=stat 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=time 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=uname 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.202Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.203Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.203Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.203Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b[62820]: ts=2024-02-10T06:22:22.203Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-10T06:22:22.457 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 bash[62759]: b250d2d9c43c27d6f36fb735b61baee03501f9569f62564506e3e0f136ebdac4 2024-02-10T06:22:22.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:22 smithi139 ceph-mon[24727]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T06:22:22.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:22 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:22.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:22 smithi139 ceph-mon[29453]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T06:22:22.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:22 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:22.874 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:22:22 smithi150 systemd[1]: Started Ceph node-exporter.b for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:22:23.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:23 smithi139 ceph-mon[24727]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T06:22:23.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:23 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:23.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:23 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:23.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:23 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:23.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:23 smithi139 ceph-mon[29453]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T06:22:23.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:23 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:23.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:23 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:23.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:23 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:23.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:23 smithi150 ceph-mon[25582]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T06:22:23.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:23 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:23.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:23 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:23.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:23 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:26 smithi150 ceph-mon[25582]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:26.503 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:26 smithi139 ceph-mon[24727]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:26.503 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:26 smithi139 ceph-mon[29453]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:27.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:27 smithi139 ceph-mon[24727]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:27.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:27.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:27.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:27.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:27 smithi139 ceph-mon[29453]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:27.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:27 smithi150 ceph-mon[25582]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:28 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:28 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:28 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:28 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.900 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:22:28] "GET /metrics HTTP/1.1" 200 37923 "" "Prometheus/2.33.4" 2024-02-10T06:22:28.900 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:28 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.900 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:28 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.900 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:28 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.900 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:28 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.901 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:28 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.901 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:28 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.901 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:28 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:28.901 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:28 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:30.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:30 smithi139 ceph-mon[24727]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:30.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:30 smithi139 ceph-mon[29453]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:30 smithi150 ceph-mon[25582]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:32.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:32.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: from='client.35273 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:32.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:32.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:32.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:32.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:32.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:31 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:32.260 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:32.260 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: from='client.35273 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:32.260 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.260 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.260 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.260 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.260 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:32.261 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:32.261 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.261 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:32.261 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:32.261 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:32.261 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:32.261 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: from='client.35273 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:32.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:32.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:32.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:32.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:32.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:32.262 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:31 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:33.288 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:32 smithi139 ceph-mon[24727]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:22:33.289 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:32 smithi139 ceph-mon[24727]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:22:33.289 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:32 smithi139 ceph-mon[29453]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:22:33.289 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:32 smithi139 ceph-mon[29453]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:22:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:32 smithi150 ceph-mon[25582]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:22:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:32 smithi150 ceph-mon[25582]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:22:34.284 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:33 smithi139 ceph-mon[24727]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:22:34.284 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:33 smithi139 ceph-mon[29453]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:22:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:33 smithi150 ceph-mon[25582]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:22:35.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:22:34.777+0000 7f9f95f8b700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:35.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Creating ceph-iscsi config... 2024-02-10T06:22:35.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:35.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:35.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:35.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:35.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:35.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:22:35.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:35.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return _run_code(code, main_globals, None, 2024-02-10T06:22:35.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:35.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: exec(code, run_globals) 2024-02-10T06:22:35.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:35.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:35.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:35.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:35.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:35.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:35.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return future.result(timeout) 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return self.__get_result() 2024-02-10T06:22:35.089 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: raise self._exception 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: config_blobs=daemon_spec.final_config, 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Creating ceph-iscsi config... 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: Traceback (most recent call last): 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:35.090 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: return _run_code(code, main_globals, None, 2024-02-10T06:22:35.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:35.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: exec(code, run_globals) 2024-02-10T06:22:35.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:35.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:35.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:35.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:35.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:35.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:35.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:35.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:35.091 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:35 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:35 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:35 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:35 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:35 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:35 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:35 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:35 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:35 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:35 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:35.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:35.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:35.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:35 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:36 smithi150 ceph-mon[25582]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:36 smithi150 ceph-mon[25582]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:22:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:36.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:36.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:22:36.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:36.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:36.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:36.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:36.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:36.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:36.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:36.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.376 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: return future.result(timeout) 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.__get_result() 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: raise self._exception 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:22:36.377 INFO:journalctl@ceph.mon.b.smithi150.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:36.378 INFO:journalctl@ceph.mon.b.smithi150.stdout: exec(code, run_globals) 2024-02-10T06:22:36.379 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:36.379 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:36.379 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:36.379 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:36.379 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:36.379 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:36.379 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:36.379 INFO:journalctl@ceph.mon.b.smithi150.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.379 INFO:journalctl@ceph.mon.b.smithi150.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:36.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:36 smithi139 ceph-mon[29453]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:36.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:36 smithi139 ceph-mon[29453]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:22:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: return future.result(timeout) 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: return self.__get_result() 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: raise self._exception 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.590 INFO:journalctl@ceph.mon.c.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:36 smithi139 ceph-mon[24727]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:36 smithi139 ceph-mon[24727]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:36.591 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T06:22:36.592 INFO:journalctl@ceph.mon.a.smithi139.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: return future.result(timeout) 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: return self.__get_result() 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: raise self._exception 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: Creating ceph-iscsi config... 2024-02-10T06:22:36.593 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: Non-zero exit code 1 from systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: Traceback (most recent call last): 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T06:22:36.594 INFO:journalctl@ceph.mon.a.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:22:36.595 INFO:journalctl@ceph.mon.a.smithi139.stdout: RuntimeError: Failed command: systemctl restart ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:22:36.595 INFO:journalctl@ceph.mon.a.smithi139.stdout: See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:22:37.354 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:37.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:37.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:38.245 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:38 smithi150 ceph-mon[25582]: Upgrade: Updating prometheus.a 2024-02-10T06:22:38.245 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:38 smithi150 ceph-mon[25582]: Deploying daemon prometheus.a on smithi150 2024-02-10T06:22:38.245 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:38 smithi150 ceph-mon[25582]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:38 smithi139 ceph-mon[24727]: Upgrade: Updating prometheus.a 2024-02-10T06:22:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:38 smithi139 ceph-mon[24727]: Deploying daemon prometheus.a on smithi150 2024-02-10T06:22:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:38 smithi139 ceph-mon[24727]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:38 smithi139 ceph-mon[29453]: Upgrade: Updating prometheus.a 2024-02-10T06:22:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:38 smithi139 ceph-mon[29453]: Deploying daemon prometheus.a on smithi150 2024-02-10T06:22:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:38 smithi139 ceph-mon[29453]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:39.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:22:38] "GET /metrics HTTP/1.1" 200 37918 "" "Prometheus/2.33.4" 2024-02-10T06:22:39.969 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 systemd[1]: Stopping Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:22:39.969 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.957Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:22:39.969 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.958Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T06:22:39.969 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.958Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T06:22:39.969 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.958Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T06:22:39.969 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.958Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T06:22:39.970 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.958Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T06:22:39.970 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.958Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T06:22:39.970 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.958Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T06:22:39.970 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.958Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T06:22:39.970 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.958Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T06:22:39.970 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.958Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T06:22:39.970 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[60742]: ts=2024-02-10T06:22:39.958Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T06:22:39.970 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:39 smithi150 podman[64166]: 2024-02-10 06:22:39.968157455 +0000 UTC m=+0.109444181 container died 7d2fef3be707e3940dd0be8127b439d807d762826cab10a6420a5a9835a87a87 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:40.251 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:40 smithi150 ceph-mon[25582]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:22:40.550 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:40 smithi150 podman[64166]: 2024-02-10 06:22:40.324428665 +0000 UTC m=+0.465715393 container cleanup 7d2fef3be707e3940dd0be8127b439d807d762826cab10a6420a5a9835a87a87 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:40.550 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:40 smithi150 bash[64166]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T06:22:40.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:40 smithi139 ceph-mon[24727]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:22:40.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:40 smithi139 ceph-mon[29453]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-10T06:22:40.748 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:22:40.874 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:40 smithi150 podman[64180]: 2024-02-10 06:22:40.550604889 +0000 UTC m=+0.584463069 container remove 7d2fef3be707e3940dd0be8127b439d807d762826cab10a6420a5a9835a87a87 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:40.874 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:40 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-10T06:22:40.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:40 smithi150 systemd[1]: Stopped Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:22:40.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:40 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Consumed 1.240s CPU time. 2024-02-10T06:22:41.270 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:40 smithi150 systemd[1]: Starting Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:22:41.270 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 podman[64313]: 2024-02-10 06:22:41.047505764 +0000 UTC m=+0.018859884 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-02-10T06:22:41.270 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 podman[64313]: 2024-02-10 06:22:41.15217622 +0000 UTC m=+0.123530383 container create 8836434edb6082a0fcd740d5fee9682f0f9bcd824cd859babfb15efd7789ec5e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:41.385 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:22:41.385 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (13m) 14s ago 21m 21.8M - 0.23.0 ba2b418f427c 5d80a4f13ac5 2024-02-10T06:22:41.385 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (13m) 14s ago 21m 54.8M - 8.3.5 dad864ee21e9 8b8f541bca1a 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (21s) 14s ago 20m 48.0M - 3.5 e1d6a67b021e 52c027f478da 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (44s) 14s ago 26m 448M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (10m) 14s ago 27m 497M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (27m) 14s ago 28m 78.2M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (26m) 14s ago 26m 57.0M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (26m) 14s ago 26m 56.7M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (26s) 14s ago 22m 7852k - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (19s) 14s ago 22m 3640k - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (25m) 14s ago 25m 54.4M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (25m) 14s ago 25m 55.1M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (25m) 14s ago 25m 53.2M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (24m) 14s ago 24m 53.5M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (24m) 14s ago 24m 55.7M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (24m) 14s ago 23m 55.2M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:22:41.386 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (23m) 14s ago 23m 52.6M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:22:41.387 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (23m) 14s ago 23m 56.2M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:22:41.387 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (50s) 14s ago 22m 60.3M - 2.33.4 514e6a882f6e 7d2fef3be707 2024-02-10T06:22:41.387 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (21m) 14s ago 21m 75.7M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:22:41.387 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (21m) 14s ago 21m 75.0M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:22:41.781 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:22:41.781 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:22:41.781 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T06:22:41.781 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:22:41.781 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:22:41.781 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:22:41.781 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:22:41.781 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:22:41.781 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:22:41.781 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:22:41.782 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": {}, 2024-02-10T06:22:41.782 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:22:41.782 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:22:41.782 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:22:41.782 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:22:41.782 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T06:22:41.782 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:22:41.782 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:22:41.782 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:22:41.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 podman[64313]: 2024-02-10 06:22:41.379138223 +0000 UTC m=+0.350492343 container init 8836434edb6082a0fcd740d5fee9682f0f9bcd824cd859babfb15efd7789ec5e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:41.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 podman[64313]: 2024-02-10 06:22:41.381587434 +0000 UTC m=+0.352941556 container start 8836434edb6082a0fcd740d5fee9682f0f9bcd824cd859babfb15efd7789ec5e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:22:41.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.404Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-10T06:22:41.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.404Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-10T06:22:41.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.404Z caller=main.go:570 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi150 (none))" 2024-02-10T06:22:41.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.404Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T06:22:41.875 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.404Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T06:22:41.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.406Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T06:22:41.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.406Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-10T06:22:41.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.408Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-10T06:22:41.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.408Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-10T06:22:41.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.410Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T06:22:41.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.411Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=641.973µs 2024-02-10T06:22:41.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.411Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T06:22:41.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.418Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-02-10T06:22:41.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.430Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-02-10T06:22:41.876 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.437Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-02-10T06:22:41.877 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.440Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-02-10T06:22:41.877 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.443Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-02-10T06:22:41.877 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.444Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-02-10T06:22:41.877 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.444Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-02-10T06:22:41.877 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.444Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.7µs wal_replay_duration=33.063434ms wbl_replay_duration=103ns total_replay_duration=33.758756ms 2024-02-10T06:22:41.877 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.447Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T06:22:41.877 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.447Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-10T06:22:41.877 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.447Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T06:22:41.877 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 bash[64313]: 8836434edb6082a0fcd740d5fee9682f0f9bcd824cd859babfb15efd7789ec5e 2024-02-10T06:22:41.877 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 systemd[1]: Started Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:22:41.877 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.465Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.468641ms db_storage=848ns remote_storage=999ns web_handler=838ns query_engine=626ns scrape=595.054µs scrape_sd=78.288µs notify=10.633µs notify_sd=14.223µs rules=17.5624ms tracing=7.786µs 2024-02-10T06:22:41.878 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.465Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-10T06:22:41.878 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:22:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:22:41.465Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-10T06:22:42.111 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:22:42.111 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:22:42.111 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:22:42.111 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-10T06:22:42.111 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [ 2024-02-10T06:22:42.111 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr" 2024-02-10T06:22:42.111 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-02-10T06:22:42.111 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "2/2 daemons upgraded", 2024-02-10T06:22:42.111 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading prometheus daemons", 2024-02-10T06:22:42.111 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:22:42.111 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:22:42.492 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:42 smithi150 ceph-mon[25582]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:42.492 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:42 smithi150 ceph-mon[25582]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:42.492 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:42 smithi150 ceph-mon[25582]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:42.492 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:42 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:42.492 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:42 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:42.492 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:42 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:42.493 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:42 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1629906942' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[24727]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[24727]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:42.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[24727]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:42.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:42.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:42.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:42.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1629906942' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[29453]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[29453]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[29453]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:42.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:42 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1629906942' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:43.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:43 smithi150 ceph-mon[25582]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:43.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:43 smithi150 ceph-mon[25582]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:43.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:43 smithi139 ceph-mon[24727]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:43.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:43 smithi139 ceph-mon[24727]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:43.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:43 smithi139 ceph-mon[29453]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:43.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:43 smithi139 ceph-mon[29453]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:22:44.507 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:44 smithi150 ceph-mon[25582]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:44.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:44 smithi139 ceph-mon[24727]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:44.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:44 smithi139 ceph-mon[29453]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:46.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:46 smithi139 ceph-mon[24727]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:46.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:46 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:46.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:46 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:46.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:46 smithi139 ceph-mon[29453]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:46.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:46 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:46.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:46 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:46 smithi150 ceph-mon[25582]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:46 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:46 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:47.511 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:47 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1549861131' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:22:47.511 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:47 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:47.511 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:47 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:47.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:47 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1549861131' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:22:47.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:47 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:47.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:47 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:47.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:47 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1549861131' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:22:47.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:47 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:47.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:47 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:48.401 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:48 smithi150 ceph-mon[25582]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:48 smithi139 ceph-mon[24727]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:48.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:48 smithi139 ceph-mon[29453]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:49.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:22:48] "GET /metrics HTTP/1.1" 200 37918 "" "Prometheus/2.43.0" 2024-02-10T06:22:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:50 smithi150 ceph-mon[25582]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:22:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:50 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:50 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:50 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:50 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:50 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:50.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[29453]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:22:50.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:50.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:50.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:50.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:50.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:50.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[24727]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:22:50.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:50.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:50.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:50.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:50.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:50 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:51.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:51 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:51 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:51 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:51.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:51 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:51.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:51 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:51.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:51 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:51.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:51 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:51.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:51 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:22:51.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:51 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:52.470 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:52 smithi139 ceph-mon[24727]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:22:52.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:52 smithi139 ceph-mon[24727]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:22:52.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:52 smithi139 ceph-mon[24727]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:52.471 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:52 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:52 smithi139 ceph-mon[29453]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:22:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:52 smithi139 ceph-mon[29453]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:22:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:52 smithi139 ceph-mon[29453]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:52 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:52.519 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:52 smithi150 ceph-mon[25582]: Reconfiguring iscsi.foo.smithi139.mtlwwr (dependencies changed)... 2024-02-10T06:22:52.519 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:52 smithi150 ceph-mon[25582]: Reconfiguring daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:22:52.519 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:52 smithi150 ceph-mon[25582]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:22:52.519 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:52 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:54.584 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:54 smithi139 ceph-mon[24727]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:54.584 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:54 smithi139 ceph-mon[29453]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:54 smithi150 ceph-mon[25582]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:22:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:22:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:22:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:22:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:22:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:22:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/4080649902' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:55.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:55.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:55 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/4080649902' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:22:55.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:22:55.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:55.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:55 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/4080649902' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:22:56.760 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:56 smithi139 ceph-mon[24727]: Upgrade: Updating alertmanager.a 2024-02-10T06:22:56.761 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:56 smithi139 ceph-mon[24727]: Deploying daemon alertmanager.a on smithi139 2024-02-10T06:22:56.761 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:56 smithi139 ceph-mon[29453]: Upgrade: Updating alertmanager.a 2024-02-10T06:22:56.761 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:56 smithi139 ceph-mon[29453]: Deploying daemon alertmanager.a on smithi139 2024-02-10T06:22:56.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:56 smithi150 ceph-mon[25582]: Upgrade: Updating alertmanager.a 2024-02-10T06:22:56.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:56 smithi150 ceph-mon[25582]: Deploying daemon alertmanager.a on smithi139 2024-02-10T06:22:57.775 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:22:57 smithi139 ceph-mon[24727]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:57.776 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:22:57 smithi139 ceph-mon[29453]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:57.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:22:57 smithi150 ceph-mon[25582]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-02-10T06:22:58.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:22:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:22:58] "GET /metrics HTTP/1.1" 200 37920 "" "Prometheus/2.43.0" 2024-02-10T06:22:59.937 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:22:59 smithi139 systemd[1]: Stopping Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:22:59.938 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:22:59 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[59133]: level=info ts=2024-02-10T06:22:59.896Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:23:00.215 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:00 smithi139 ceph-mon[24727]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:23:00.215 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:22:59 smithi139 podman[79932]: 2024-02-10 06:22:59.937547428 +0000 UTC m=+0.133740717 container died 5d80a4f13ac5cc64dc74459a8e7676a8f7bfd7fec3c9fb8445ddb40722517003 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:23:00.215 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:00 smithi139 ceph-mon[29453]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:23:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:00 smithi150 ceph-mon[25582]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:23:00.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:00 smithi139 podman[79932]: 2024-02-10 06:23:00.397094785 +0000 UTC m=+0.593288067 container cleanup 5d80a4f13ac5cc64dc74459a8e7676a8f7bfd7fec3c9fb8445ddb40722517003 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:23:00.586 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:00 smithi139 bash[79932]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T06:23:00.952 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:00 smithi139 podman[79946]: 2024-02-10 06:23:00.690219784 +0000 UTC m=+0.748027202 container remove 5d80a4f13ac5cc64dc74459a8e7676a8f7bfd7fec3c9fb8445ddb40722517003 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:23:00.952 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:00 smithi139 podman[79946]: 2024-02-10 06:23:00.765870366 +0000 UTC m=+0.823677786 volume remove 93e03c5c68b1eba78c9403fddb28c2a5c0a93320ec025b6ce54411a3088fa1e9 2024-02-10T06:23:01.336 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:00 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T06:23:01.336 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:00 smithi139 systemd[1]: Stopped Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:23:01.336 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:00 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Consumed 1.503s CPU time. 2024-02-10T06:23:01.337 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 systemd[1]: Starting Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:23:01.717 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 podman[80078]: 2024-02-10 06:23:01.283038666 +0000 UTC m=+0.017723880 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-10T06:23:01.717 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 podman[80078]: 2024-02-10 06:23:01.408993935 +0000 UTC m=+0.143679149 volume create b00620828cc3d4f2d7de533b8940b2e78eeda555a0d9679e0c54feb57a9cccfe 2024-02-10T06:23:01.717 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 podman[80078]: 2024-02-10 06:23:01.509431142 +0000 UTC m=+0.244116355 container create 2b6a81902a4c8ccf4e1af789dac614e788d9656f4b3b3356f22c4541ed8b80e0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:23:02.086 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 podman[80078]: 2024-02-10 06:23:01.84555524 +0000 UTC m=+0.580240454 container init 2b6a81902a4c8ccf4e1af789dac614e788d9656f4b3b3356f22c4541ed8b80e0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:23:02.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 podman[80078]: 2024-02-10 06:23:01.848338413 +0000 UTC m=+0.583023634 container start 2b6a81902a4c8ccf4e1af789dac614e788d9656f4b3b3356f22c4541ed8b80e0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:23:02.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:23:01.860Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-10T06:23:02.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:23:01.860Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-10T06:23:02.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:23:01.861Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.139 port=9094 2024-02-10T06:23:02.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:23:01.862Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T06:23:02.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:23:01.885Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T06:23:02.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:23:01.886Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T06:23:02.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:23:01.889Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-10T06:23:02.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:23:01.889Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-10T06:23:02.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 bash[80078]: 2b6a81902a4c8ccf4e1af789dac614e788d9656f4b3b3356f22c4541ed8b80e0 2024-02-10T06:23:02.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:01 smithi139 systemd[1]: Started Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:23:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:02 smithi150 ceph-mon[25582]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:23:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:02 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:02 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:02 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:02.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:02 smithi139 ceph-mon[29453]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:23:02.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:02 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:02.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:02 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:02.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:02 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:02.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:02 smithi139 ceph-mon[24727]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:23:02.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:02 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:02.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:02 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:02.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:02 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:04.087 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:23:03.863Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000552299s 2024-02-10T06:23:04.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:04 smithi150 ceph-mon[25582]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:23:04.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:04 smithi139 ceph-mon[24727]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:23:04.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:04 smithi139 ceph-mon[29453]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:23:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:06 smithi150 ceph-mon[25582]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:23:06.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:06 smithi150 ceph-mon[25582]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:06.428 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:06 smithi139 ceph-mon[24727]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:23:06.428 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:06 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:06.429 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:06 smithi139 ceph-mon[29453]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:23:06.429 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:06 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:07.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:07.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[29453]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:23:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[24727]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:23:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:07 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:07 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:07 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:07 smithi150 ceph-mon[25582]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:23:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:07 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:07.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:07 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:07.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:07 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:09.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:08 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:09.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:08 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:09.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:08 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:09.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:08 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:09.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:23:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:23:08] "GET /metrics HTTP/1.1" 200 37921 "" "Prometheus/2.43.0" 2024-02-10T06:23:09.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:08 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:09.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:08 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:10 smithi139 ceph-mon[24727]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:23:10.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:10 smithi139 ceph-mon[29453]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:23:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:10 smithi150 ceph-mon[25582]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:23:12.116 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:23:11 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:23:11.865Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002949329s 2024-02-10T06:23:12.116 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:12.116 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:12.117 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:12.117 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:12.117 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:12.117 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:12.117 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:23:12.117 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:12.342 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:12.342 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.450 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:23:12.451 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.451 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.451 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.451 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.451 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.451 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.451 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.451 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.451 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.452 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.453 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.453 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.453 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.453 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.453 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.453 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.453 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.453 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:12.453 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:13.107 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:23:13.107 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (11s) 6s ago 22m 16.7M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:23:13.107 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (13m) 6s ago 22m 53.8M - 8.3.5 dad864ee21e9 8b8f541bca1a 2024-02-10T06:23:13.107 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (19s) 6s ago 21m 50.2M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:23:13.107 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (76s) 6s ago 26m 449M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:23:13.107 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (11m) 6s ago 28m 503M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:23:13.107 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (28m) 6s ago 28m 75.3M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (27m) 6s ago 27m 53.5M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (27m) 6s ago 27m 54.4M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (57s) 6s ago 23m 8376k - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (51s) 6s ago 22m 15.6M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (26m) 6s ago 26m 54.5M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (26m) 6s ago 26m 55.1M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (25m) 6s ago 25m 53.3M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (25m) 6s ago 25m 53.5M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (24m) 6s ago 24m 55.7M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (24m) 6s ago 24m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (24m) 6s ago 24m 52.6M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (23m) 6s ago 23m 56.2M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:23:13.108 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (32s) 6s ago 23m 47.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:23:13.109 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (21m) 6s ago 21m 75.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:23:13.109 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (21m) 6s ago 21m 75.1M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:23:13.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:13 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:23:13.492 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:23:13.492 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:23:13.492 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T06:23:13.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:23:13.492 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:23:13.492 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:23:13.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:23:13.492 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:23:13.492 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:23:13.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:23:13.493 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": {}, 2024-02-10T06:23:13.493 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:23:13.493 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:23:13.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:23:13.493 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:23:13.493 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T06:23:13.493 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:23:13.493 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:23:13.493 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:23:13.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:13 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:23:13.493 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:13 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:23:13.818 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:23:13.818 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:23:13.818 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:23:13.818 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-10T06:23:13.818 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [ 2024-02-10T06:23:13.819 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr" 2024-02-10T06:23:13.819 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-02-10T06:23:13.819 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "2/2 daemons upgraded", 2024-02-10T06:23:13.819 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading grafana daemons", 2024-02-10T06:23:13.819 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:23:13.819 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:23:14.398 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[24727]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:14.398 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[24727]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[24727]: from='client.25573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[24727]: Upgrade: Updating grafana.a 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[24727]: Deploying daemon grafana.a on smithi150 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[24727]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1865453113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[29453]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[29453]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[29453]: from='client.25573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[29453]: Upgrade: Updating grafana.a 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[29453]: Deploying daemon grafana.a on smithi150 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[29453]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:14.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:14 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1865453113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:14.512 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:14 smithi150 ceph-mon[25582]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:14.512 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:14 smithi150 ceph-mon[25582]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:14.512 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:14 smithi150 ceph-mon[25582]: from='client.25573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:14.512 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:14 smithi150 ceph-mon[25582]: Upgrade: Updating grafana.a 2024-02-10T06:23:14.512 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:14 smithi150 ceph-mon[25582]: Deploying daemon grafana.a on smithi150 2024-02-10T06:23:14.512 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:14 smithi150 ceph-mon[25582]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:14.512 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:14 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1865453113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:15.512 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:15 smithi150 ceph-mon[25582]: from='client.25591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:15.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:15 smithi139 ceph-mon[24727]: from='client.25591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:15.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:15 smithi139 ceph-mon[29453]: from='client.25591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:16.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:15 smithi150 systemd[1]: Stopping Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:23:16.341 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:16 smithi150 ceph-mon[25582]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:16.341 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:16 smithi150 ceph-mon[25582]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:16.341 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:16 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[49699]: t=2024-02-10T06:23:16+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-10T06:23:16.341 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:16 smithi150 podman[66283]: 2024-02-10 06:23:16.066714051 +0000 UTC m=+0.130674802 container died 8b8f541bca1a322ef633037fcd2901c35335521e6066517996bc5f6daf198afe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, vendor=Red Hat, Inc., release=236.1648460182, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, distribution-scope=public, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-02-10T06:23:16.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:16 smithi139 ceph-mon[29453]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:16.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:16 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:16.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:16 smithi139 ceph-mon[24727]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:16.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:16 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:16.624 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:16 smithi150 podman[66283]: 2024-02-10 06:23:16.414401007 +0000 UTC m=+0.478361768 container cleanup 8b8f541bca1a322ef633037fcd2901c35335521e6066517996bc5f6daf198afe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, io.openshift.tags=base rhel8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, io.buildah.version=1.24.2, name=ubi8, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., release=236.1648460182, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-02-10T06:23:16.624 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:16 smithi150 bash[66283]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a 2024-02-10T06:23:17.124 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:16 smithi150 podman[66297]: 2024-02-10 06:23:16.832480801 +0000 UTC m=+0.763011556 container remove 8b8f541bca1a322ef633037fcd2901c35335521e6066517996bc5f6daf198afe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, architecture=x86_64, maintainer=Paul Cuzner , vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, name=ubi8, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, io.buildah.version=1.24.2, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5) 2024-02-10T06:23:17.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:16 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Deactivated successfully. 2024-02-10T06:23:17.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:16 smithi150 systemd[1]: Stopped Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:23:17.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:16 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Consumed 5.464s CPU time. 2024-02-10T06:23:17.523 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 systemd[1]: Starting Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:23:17.523 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 podman[66428]: 2024-02-10 06:23:17.29569518 +0000 UTC m=+0.017336725 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-02-10T06:23:17.523 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 podman[66428]: 2024-02-10 06:23:17.409026116 +0000 UTC m=+0.130667649 container create dae5565f9ce7d389d08ca511fd0f40ac271dab56d3089df2e6da8c5367e4b8b3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, version=8.5, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, architecture=x86_64, description=Ceph Grafana Container, vcs-type=git, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897) 2024-02-10T06:23:17.775 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 podman[66428]: 2024-02-10 06:23:17.62010603 +0000 UTC m=+0.341747564 container init dae5565f9ce7d389d08ca511fd0f40ac271dab56d3089df2e6da8c5367e4b8b3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, distribution-scope=public, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, vcs-type=git, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., architecture=x86_64) 2024-02-10T06:23:17.775 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 podman[66428]: 2024-02-10 06:23:17.622761312 +0000 UTC m=+0.344402853 container start dae5565f9ce7d389d08ca511fd0f40ac271dab56d3089df2e6da8c5367e4b8b3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, io.openshift.tags=base rhel8, distribution-scope=public, io.openshift.expose-services=, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, name=ubi8, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-02-10T06:23:17.775 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 bash[66428]: dae5565f9ce7d389d08ca511fd0f40ac271dab56d3089df2e6da8c5367e4b8b3 2024-02-10T06:23:17.775 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 systemd[1]: Started Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:23:18.046 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.775101539Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-02-10T06:23:18.046 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.775300961Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.775318371Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.775325718Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.775331775Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.775337613Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.775343581Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.77535033Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.775356424Z level=info msg="Path Home" path=/usr/share/grafana 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.775362103Z level=info msg="Path Data" path=/var/lib/grafana 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.77536746Z level=info msg="Path Logs" path=/var/log/grafana 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.77537278Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.775378311Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=settings t=2024-02-10T06:23:17.775384392Z level=info msg="App mode production" 2024-02-10T06:23:18.047 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=sqlstore t=2024-02-10T06:23:17.775451148Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-02-10T06:23:18.048 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=sqlstore t=2024-02-10T06:23:17.775475823Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-10T06:23:18.048 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.786898959Z level=info msg="Starting DB migrations" 2024-02-10T06:23:18.048 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.790301091Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-02-10T06:23:18.048 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.798957587Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.799992429Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.800385395Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.801539139Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.802559025Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.803523316Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.804375052Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.804853953Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.805776768Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.806614373Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.807627548Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.808487187Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-02-10T06:23:18.049 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.809493367Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.812324181Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.815391684Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.818239854Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.820961488Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.823717234Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.826602669Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.829356788Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.832174148Z level=info msg="Executing migration" id="create provenance_type table" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.834973002Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.8376199Z level=info msg="Executing migration" id="create alert_image table" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.840458223Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.843293845Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.845997314Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-02-10T06:23:18.050 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.84881755Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-02-10T06:23:18.051 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.851693645Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-02-10T06:23:18.051 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.851807236Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-02-10T06:23:18.051 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.854560706Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-02-10T06:23:18.051 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.857314851Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-02-10T06:23:18.051 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.85994934Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-02-10T06:23:18.051 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.862616297Z level=info msg="Executing migration" id="create secrets table" 2024-02-10T06:23:18.051 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.865268157Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-02-10T06:23:18.051 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.870007262Z level=info msg="Executing migration" id="add name column into data_keys" 2024-02-10T06:23:18.051 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.87256425Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-02-10T06:23:18.051 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.875089662Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-02-10T06:23:18.051 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.879781538Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.884532436Z level=info msg="Executing migration" id="add column hidden to role table" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.887201178Z level=info msg="Executing migration" id="create query_history table v1" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.889819005Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.892388424Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.895094198Z level=info msg="Executing migration" id="teams permissions migration" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.897794265Z level=info msg="Executing migration" id="dashboard permissions" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.899102299Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.899939192Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.900759529Z level=info msg="Executing migration" id="alerting notification permissions" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.901578455Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.902474683Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.903304369Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.904380343Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-02-10T06:23:18.052 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.905208416Z level=info msg="Executing migration" id="create correlation table v1" 2024-02-10T06:23:18.053 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.90603514Z level=info msg="Executing migration" id="add index correlations.uid" 2024-02-10T06:23:18.053 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.906808122Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-02-10T06:23:18.053 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.90766657Z level=info msg="Executing migration" id="add correlation config column" 2024-02-10T06:23:18.053 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.908699087Z level=info msg="Executing migration" id="create entity_events table" 2024-02-10T06:23:18.053 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.909579871Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-02-10T06:23:18.053 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.910427307Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.910536384Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.911255692Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.911363411Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.911735522Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.912276538Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.912709651Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.913566609Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.914389749Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.915258129Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.916098146Z level=info msg="Executing migration" id="Drop public config table" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.916960479Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.917836654Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.918689959Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.919545584Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-02-10T06:23:18.054 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.920404573Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.923486825Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.924548887Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.925584588Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.926402275Z level=info msg="Executing migration" id="add share column" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.927544332Z level=info msg="Executing migration" id="create default alerting folders" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.928354625Z level=info msg="Executing migration" id="create file table" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.928815197Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.929292001Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.929753122Z level=info msg="Executing migration" id="create file_meta table" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.9301844Z level=info msg="Executing migration" id="file table idx: path key" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.93065116Z level=info msg="Executing migration" id="set path collation in file table" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.930942917Z level=info msg="Executing migration" id="managed permissions migration" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.932062129Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.932708051Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.933246722Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.934306344Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-02-10T06:23:18.055 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.934650058Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.93514509Z level=info msg="Executing migration" id="update group index for alert rules" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.935578039Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.93611329Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.93651711Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.937580029Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.938640089Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.939493118Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.949392596Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.949888501Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.950441947Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.953975792Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.956665288Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-02-10T06:23:18.056 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.95705638Z level=info msg="Executing migration" id="create folder table" 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.957488353Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.958130295Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.958596562Z level=info msg="Executing migration" id="Update folder title length" 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.958887421Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=migrator t=2024-02-10T06:23:17.959398361Z level=info msg="migrations completed" performed=105 skipped=376 duration=169.130341ms 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=sqlstore t=2024-02-10T06:23:17.959750154Z level=info msg="Created default organization" 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=plugin.loader t=2024-02-10T06:23:18.018414051Z level=info msg="Plugin registered" pluginID=input 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=plugin.loader t=2024-02-10T06:23:18.045839583Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=plugin.loader t=2024-02-10T06:23:18.045870983Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=secrets t=2024-02-10T06:23:18.045913541Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=query_data t=2024-02-10T06:23:18.048287856Z level=info msg="Query Service initialization" 2024-02-10T06:23:18.057 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=live.push_http t=2024-02-10T06:23:18.050844468Z level=info msg="Live Push Gateway initialization" 2024-02-10T06:23:18.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-mon[25582]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:18.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:18.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:18.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:18 smithi139 ceph-mon[24727]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:18 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:18 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:18 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:18.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:18 smithi139 ceph-mon[29453]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:18.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:18 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:18.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:18 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:18.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:18 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:19.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:23:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:23:18] "GET /metrics HTTP/1.1" 200 37921 "" "Prometheus/2.43.0" 2024-02-10T06:23:19.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=infra.usagestats.collector t=2024-02-10T06:23:18.710932054Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-02-10T06:23:19.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=provisioning.datasources t=2024-02-10T06:23:18.712719603Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-02-10T06:23:19.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=provisioning.datasources t=2024-02-10T06:23:18.712739088Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-02-10T06:23:19.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=provisioning.datasources t=2024-02-10T06:23:18.712889751Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-10T06:23:19.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=provisioning.alerting t=2024-02-10T06:23:18.736174211Z level=info msg="starting to provision alerting" 2024-02-10T06:23:19.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=provisioning.alerting t=2024-02-10T06:23:18.736202349Z level=info msg="finished to provision alerting" 2024-02-10T06:23:19.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=grafanaStorageLogger t=2024-02-10T06:23:18.736333561Z level=info msg="storage starting" 2024-02-10T06:23:19.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=ngalert.state.manager t=2024-02-10T06:23:18.736603603Z level=info msg="Warming state cache for startup" 2024-02-10T06:23:19.125 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=ngalert.state.manager t=2024-02-10T06:23:18.736764245Z level=info msg="State cache has been initialized" states=0 duration=160.049µs 2024-02-10T06:23:19.126 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=ticker t=2024-02-10T06:23:18.736870344Z level=info msg=starting first_tick=2024-02-10T06:23:20Z 2024-02-10T06:23:19.126 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=ngalert.multiorg.alertmanager t=2024-02-10T06:23:18.736921575Z level=info msg="starting MultiOrg Alertmanager" 2024-02-10T06:23:19.126 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=http.server t=2024-02-10T06:23:18.738928312Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T06:23:19.126 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=sqlstore.transactions t=2024-02-10T06:23:18.765894094Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-10T06:23:19.126 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:23:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=sqlstore.transactions t=2024-02-10T06:23:18.777516276Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-02-10T06:23:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:20 smithi150 ceph-mon[25582]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:20.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:20 smithi139 ceph-mon[24727]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:20.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:20 smithi139 ceph-mon[29453]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:22.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[29453]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:22.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:22.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:22.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:22.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:22.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:22.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[24727]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:22.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:22.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:22.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:22.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:22.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:22 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:22 smithi150 ceph-mon[25582]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:22 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:22 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:22 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:22 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:22 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:24.077 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:24 smithi139 ceph-mon[24727]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:24.078 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:24 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:24.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:24 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:24.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:24 smithi139 ceph-mon[29453]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:24.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:24 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:24.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:24 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:24.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:24 smithi150 ceph-mon[25582]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:24.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:24 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:24.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:24 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:26 smithi150 ceph-mon[25582]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:26.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:26 smithi150 ceph-mon[25582]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:26 smithi139 ceph-mon[24727]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:26 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:26.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:26 smithi139 ceph-mon[29453]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:26.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:26 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:27.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:27.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:27.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:27.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:23:27.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: Upgrade: Finalizing container_image settings 2024-02-10T06:23:27.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:23:27.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:23:27.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:23:27.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:27.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:27.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: Upgrade: Finalizing container_image settings 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:23:27.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:23:27.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:23:27.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:23:27.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:23:27.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:23:27.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:23:27.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:23:27.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:23:27.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:23:27.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:23:27.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:23:27.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:23:27.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:23:27.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:23:27.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:23:27.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:23:27.845 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:23:27.845 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:23:27.845 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:23:27.845 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:23:27.845 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.845 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: Upgrade: Complete! 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:27.846 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:27.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:27.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:27.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:27.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:27.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:23:27.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:23:27.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:23:27.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: Upgrade: Complete! 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:23:27.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:23:27.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:23:27.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:27.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:27.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:27.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:27.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:27.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:27.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: Upgrade: Finalizing container_image settings 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:23:27.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:23:27.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: Upgrade: Complete! 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:27.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:27.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:27.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:27.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:27 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:29.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:23:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:23:28] "GET /metrics HTTP/1.1" 200 37922 "" "Prometheus/2.43.0" 2024-02-10T06:23:30.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:30 smithi150 ceph-mon[25582]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:30.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:30 smithi139 ceph-mon[24727]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:30.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:30 smithi139 ceph-mon[29453]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:32.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:32 smithi150 ceph-mon[25582]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:32 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:32.388 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:32 smithi139 ceph-mon[29453]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:32.388 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:32 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:32.389 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:32 smithi139 ceph-mon[24727]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:32.389 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:32 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:23:34.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:34 smithi150 ceph-mon[25582]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:34.546 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:34 smithi139 ceph-mon[24727]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:34.546 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:34 smithi139 ceph-mon[29453]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:36.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:36 smithi139 ceph-mon[24727]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:36 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:36 smithi139 ceph-mon[29453]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:36 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:36.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:36 smithi150 ceph-mon[25582]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:36.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:36 smithi150 ceph-mon[25582]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:37.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:37.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:37.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:37 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:38 smithi139 ceph-mon[24727]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:38 smithi139 ceph-mon[29453]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:38 smithi150 ceph-mon[25582]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:39.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:23:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:23:38] "GET /metrics HTTP/1.1" 200 37920 "" "Prometheus/2.43.0" 2024-02-10T06:23:40.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:40 smithi139 ceph-mon[24727]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:40.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:40 smithi139 ceph-mon[29453]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:40.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:40 smithi150 ceph-mon[25582]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:42 smithi139 ceph-mon[24727]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:42 smithi139 ceph-mon[29453]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:42.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:42 smithi150 ceph-mon[25582]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:44 smithi139 ceph-mon[24727]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:44 smithi139 ceph-mon[29453]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:44 smithi150 ceph-mon[25582]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:44.945 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-02-10T06:23:45.259 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:45 smithi139 ceph-mon[24727]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:45.259 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:45 smithi139 ceph-mon[29453]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:45.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:45 smithi150 ceph-mon[25582]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:46.211 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:46 smithi139 ceph-mon[24727]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:46.211 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:46 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:46.212 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:46 smithi139 ceph-mon[29453]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:46.212 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:46 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:46 smithi150 ceph-mon[25582]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:46 smithi150 ceph-mon[25582]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:46.973 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:23:47.518 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:47 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1347910150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:47.518 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:47 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1347910150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:47.541 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-02-10T06:23:47.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:47 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1347910150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:48 smithi139 ceph-mon[24727]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:48 smithi139 ceph-mon[29453]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:48 smithi150 ceph-mon[25582]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:48.992 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:23:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:23:48] "GET /metrics HTTP/1.1" 200 37920 "" "Prometheus/2.43.0" 2024-02-10T06:23:49.625 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)" 2024-02-10T06:23:50.265 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:50 smithi139 ceph-mon[24727]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:50.265 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:50 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2801668883' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:50.266 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:50 smithi139 ceph-mon[29453]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:50.266 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:50 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2801668883' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:50.290 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-02-10T06:23:50.427 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:50 smithi150 ceph-mon[25582]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:50.427 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:50 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/2801668883' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:52 smithi139 ceph-mon[24727]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:52.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:52 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:52 smithi139 ceph-mon[29453]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:52 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:52.453 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:23:52.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:52 smithi150 ceph-mon[25582]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:52.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:52 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:53.112 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-02-10T06:23:53.421 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:53 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/158849107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:53.421 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:53 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/158849107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:53.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:53 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/158849107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:23:54.434 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:54 smithi139 ceph-mon[24727]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:54.434 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:54 smithi139 ceph-mon[29453]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:54 smithi150 ceph-mon[25582]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:56.276 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:56 smithi139 ceph-mon[24727]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:56.276 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:56 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:56.276 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:56 smithi139 ceph-mon[24727]: from='client.25621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:56.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:56 smithi139 ceph-mon[29453]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:56.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:56 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:56.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:56 smithi139 ceph-mon[29453]: from='client.25621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:56 smithi150 ceph-mon[25582]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:56 smithi150 ceph-mon[25582]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:56.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:56 smithi150 ceph-mon[25582]: from='client.25621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:23:57.184 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:23:57.878 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade status' 2024-02-10T06:23:58.516 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:23:58 smithi139 ceph-mon[24727]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:58.516 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:23:58 smithi139 ceph-mon[29453]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:23:58 smithi150 ceph-mon[25582]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:58.794 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:23:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:23:58] "GET /metrics HTTP/1.1" 200 37932 "" "Prometheus/2.43.0" 2024-02-10T06:23:59.827 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:23:59.828 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": null, 2024-02-10T06:23:59.828 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": false, 2024-02-10T06:23:59.828 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "", 2024-02-10T06:23:59.828 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:23:59.828 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": null, 2024-02-10T06:23:59.828 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "", 2024-02-10T06:23:59.828 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:23:59.828 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:24:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:00 smithi150 ceph-mon[25582]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:00.401 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:00 smithi139 ceph-mon[24727]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:00.401 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:00 smithi139 ceph-mon[29453]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:00.427 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T06:24:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:01 smithi139 ceph-mon[24727]: from='client.25627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:01.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:01 smithi139 ceph-mon[29453]: from='client.25627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:01 smithi150 ceph-mon[25582]: from='client.25627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:02.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:02 smithi139 ceph-mon[24727]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:02.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:02 smithi139 ceph-mon[29453]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:02.446 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_OK 2024-02-10T06:24:02.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:02 smithi150 ceph-mon[25582]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:03.112 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '"'"'{print $2}'"'"')' 2024-02-10T06:24:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:03 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1959010357' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:24:03.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:03 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1959010357' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:24:03.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:03 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/1959010357' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:24:04.444 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:04 smithi139 ceph-mon[24727]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:04.444 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:04 smithi139 ceph-mon[29453]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:04 smithi150 ceph-mon[25582]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:06.479 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:06 smithi139 ceph-mon[24727]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:06.480 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:06 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:06.480 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:06 smithi139 ceph-mon[29453]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:06.480 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:06 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:06 smithi150 ceph-mon[25582]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:06 smithi150 ceph-mon[25582]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:07.577 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:07 smithi139 ceph-mon[24727]: from='client.25639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:07.577 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:07 smithi139 ceph-mon[24727]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mon", "hosts": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:07.577 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:07 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:07.577 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:07 smithi139 ceph-mon[29453]: from='client.25639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:07.577 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:07 smithi139 ceph-mon[29453]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mon", "hosts": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:07.578 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:07 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:07.611 INFO:teuthology.orchestra.run.smithi139.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:07 smithi150 ceph-mon[25582]: from='client.25639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:07 smithi150 ceph-mon[25582]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mon", "hosts": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:07 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:08.264 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:08 smithi139 ceph-mon[24727]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:08.264 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:08 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:08.264 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:08 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:08.265 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:08 smithi139 ceph-mon[29453]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:08.265 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:08 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:08.265 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:08 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:08.287 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T06:24:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:08 smithi150 ceph-mon[25582]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:08.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:08 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:08.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:08 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:08.892 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:24:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:24:08] "GET /metrics HTTP/1.1" 200 37930 "" "Prometheus/2.43.0" 2024-02-10T06:24:09.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:09 smithi139 ceph-mon[29453]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:09.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:09 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:09.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:09 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:09.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:09 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:09.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:09 smithi139 ceph-mon[24727]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:09.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:09 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:09.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:09 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:09.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:09 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:09.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:09 smithi150 ceph-mon[25582]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:09.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:09 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:09.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:09 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:09.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:09 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:10.319 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:10 smithi139 ceph-mon[24727]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:10.319 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:10 smithi139 ceph-mon[24727]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:10.320 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:10 smithi139 ceph-mon[29453]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:10.320 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:10 smithi139 ceph-mon[29453]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:10.591 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:24:10.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:10 smithi150 ceph-mon[25582]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:10.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:10 smithi150 ceph-mon[25582]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:11.246 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:24:11.246 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (69s) 49s ago 23m 16.8M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (53s) 49s ago 23m 51.7M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (77s) 49s ago 22m 50.3M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (2m) 49s ago 27m 449M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (12m) 49s ago 29m 504M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (29m) 49s ago 29m 76.5M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (27m) 49s ago 27m 54.3M 2048M 17.2.0 e1d6a67b021e 51b65c6e04ce 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (28m) 49s ago 28m 55.2M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (115s) 49s ago 23m 14.9M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (109s) 49s ago 23m 15.7M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (27m) 49s ago 27m 54.5M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (27m) 49s ago 27m 55.1M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (26m) 49s ago 26m 53.3M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (26m) 49s ago 26m 53.5M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:24:11.247 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (25m) 49s ago 25m 55.7M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:24:11.248 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (25m) 49s ago 25m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:24:11.248 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (25m) 49s ago 25m 52.6M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:24:11.248 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (24m) 49s ago 24m 56.2M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:24:11.248 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (90s) 49s ago 24m 47.2M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:24:11.248 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (22m) 49s ago 22m 75.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:24:11.248 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (22m) 49s ago 22m 75.2M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:24:11.644 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": {}, 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:24:11.645 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:24:11.646 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:24:11.968 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:24:11.968 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:24:11.968 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:24:11.969 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi150", 2024-02-10T06:24:11.969 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:24:11.969 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "0/1 daemons upgraded", 2024-02-10T06:24:11.969 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading mon daemons", 2024-02-10T06:24:11.969 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:24:11.969 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:24:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[24727]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[24727]: from='client.25651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[24727]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:24:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-10T06:24:12.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/3218673519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:12.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[29453]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:12.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[29453]: from='client.25651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:12.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[29453]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:12.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:12.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:12.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:12.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:12.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:24:12.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-10T06:24:12.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:12 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/3218673519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:12.432 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:12 smithi150 ceph-mon[25582]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:12.432 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:12 smithi150 ceph-mon[25582]: from='client.25651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:12.432 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:12 smithi150 ceph-mon[25582]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:12.432 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:12 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:12.432 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:12.432 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:12.432 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:12 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:12.432 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:24:12.433 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:12 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-10T06:24:12.433 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:12 smithi150 ceph-mon[25582]: from='client.? 172.21.15.139:0/3218673519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:13.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[24727]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:13.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[24727]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:24:13.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[24727]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:24:13.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all mgr 2024-02-10T06:24:13.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[24727]: Upgrade: It appears safe to stop mon.b 2024-02-10T06:24:13.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[24727]: from='client.25675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:13.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[29453]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:13.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[29453]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:24:13.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[29453]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:24:13.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all mgr 2024-02-10T06:24:13.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[29453]: Upgrade: It appears safe to stop mon.b 2024-02-10T06:24:13.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:13 smithi139 ceph-mon[29453]: from='client.25675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:13 smithi150 ceph-mon[25582]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:13 smithi150 ceph-mon[25582]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:24:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:13 smithi150 ceph-mon[25582]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:24:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:13 smithi150 ceph-mon[25582]: Upgrade: Setting container_image for all mgr 2024-02-10T06:24:13.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:13 smithi150 ceph-mon[25582]: Upgrade: It appears safe to stop mon.b 2024-02-10T06:24:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:13 smithi150 ceph-mon[25582]: from='client.25675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:14.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:14 smithi150 ceph-mon[25582]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:14.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:14 smithi150 ceph-mon[25582]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:14.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:14 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:24:14.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:14 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:24:14.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:14 smithi150 ceph-mon[25582]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:14.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:14 smithi139 ceph-mon[24727]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:14.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:14 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:14.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:14 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:24:14.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:14 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:24:14.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:14 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:14 smithi139 ceph-mon[29453]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:14 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:14 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:24:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:14 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:24:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:14 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:15.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:15 smithi150 systemd[1]: Stopping Ceph mon.b for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:24:15.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:15 smithi139 ceph-mon[24727]: Upgrade: Updating mon.b 2024-02-10T06:24:15.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:15 smithi139 ceph-mon[24727]: Deploying daemon mon.b on smithi150 2024-02-10T06:24:15.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:15 smithi139 ceph-mon[29453]: Upgrade: Updating mon.b 2024-02-10T06:24:15.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:15 smithi139 ceph-mon[29453]: Deploying daemon mon.b on smithi150 2024-02-10T06:24:15.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:15 smithi150 ceph-mon[25582]: Upgrade: Updating mon.b 2024-02-10T06:24:15.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:15 smithi150 ceph-mon[25582]: Deploying daemon mon.b on smithi150 2024-02-10T06:24:15.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b[25565]: 2024-02-10T06:24:15.263+0000 7f6736be8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:24:15.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b[25565]: 2024-02-10T06:24:15.263+0000 7f6736be8700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-10T06:24:15.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:15 smithi150 podman[67891]: 2024-02-10 06:24:15.387308102 +0000 UTC m=+0.242980945 container died 51b65c6e04ce76ace48f94651a265b2a5ac31b0c55474b3633996e1cfe795eed (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, RELEASE=HEAD, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git) 2024-02-10T06:24:16.008 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:15 smithi150 podman[67891]: 2024-02-10 06:24:15.739756159 +0000 UTC m=+0.595429003 container cleanup 51b65c6e04ce76ace48f94651a265b2a5ac31b0c55474b3633996e1cfe795eed (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, release=754, version=8, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, name=centos-stream, architecture=x86_64) 2024-02-10T06:24:16.008 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:15 smithi150 bash[67891]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b 2024-02-10T06:24:16.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 podman[67905]: 2024-02-10 06:24:16.007847839 +0000 UTC m=+0.625029478 container remove 51b65c6e04ce76ace48f94651a265b2a5ac31b0c55474b3633996e1cfe795eed (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , version=8, release=754, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.k8s.display-name=CentOS Stream 8, distribution-scope=public) 2024-02-10T06:24:16.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.b.service: Deactivated successfully. 2024-02-10T06:24:16.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 systemd[1]: Stopped Ceph mon.b for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:24:16.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.b.service: Consumed 18.021s CPU time. 2024-02-10T06:24:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 systemd[1]: Starting Ceph mon.b for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:24:16.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 podman[68037]: 2024-02-10 06:24:16.575449521 +0000 UTC m=+0.110718029 container create ec59077c047e2307507159003ccdb94a9ba436e549d632b318e582e84428ac72 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0) 2024-02-10T06:24:16.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 podman[68037]: 2024-02-10 06:24:16.483207138 +0000 UTC m=+0.018475600 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 podman[68037]: 2024-02-10 06:24:16.845180483 +0000 UTC m=+0.380448975 container init ec59077c047e2307507159003ccdb94a9ba436e549d632b318e582e84428ac72 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 podman[68037]: 2024-02-10 06:24:16.84847013 +0000 UTC m=+0.383738597 container start ec59077c047e2307507159003ccdb94a9ba436e549d632b318e582e84428ac72 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0) 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: pidfile_write: ignore empty --pid-file 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: load: jerasure load: lrc 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: RocksDB version: 7.9.2 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Git sha 0 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: DB SUMMARY 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: DB Session ID: 8K94NEGMROMXBOCV6JX1 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: CURRENT file: CURRENT 2024-02-10T06:24:16.975 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: MANIFEST file: MANIFEST-000009 size: 2814 Bytes 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000054.sst 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000052.log size: 2704736 ; 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.error_if_exists: 0 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.create_if_missing: 0 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.paranoid_checks: 1 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.env: 0x55a2ae0ccae0 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.info_log: 0x55a2b05118c0 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T06:24:16.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.statistics: (nil) 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.use_fsync: 0 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_log_file_size: 0 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.allow_fallocate: 1 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.use_direct_reads: 0 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.db_log_dir: 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.wal_dir: 2024-02-10T06:24:16.977 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.write_buffer_manager: 0x55a2b04e4dc0 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T06:24:16.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T06:24:16.979 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T06:24:16.979 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.unordered_write: 0 2024-02-10T06:24:16.979 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T06:24:16.979 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T06:24:16.979 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T06:24:16.979 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T06:24:16.979 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.row_cache: None 2024-02-10T06:24:16.979 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.wal_filter: None 2024-02-10T06:24:16.979 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T06:24:16.980 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T06:24:16.980 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.two_write_queues: 0 2024-02-10T06:24:16.980 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T06:24:16.980 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.wal_compression: 0 2024-02-10T06:24:16.980 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.atomic_flush: 0 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.log_readahead_size: 0 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_background_jobs: 2 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_background_compactions: -1 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_subcompactions: 1 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T06:24:16.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_open_files: -1 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_background_flushes: -1 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Compression algorithms supported: 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T06:24:16.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: kZSTD supported: 0 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: kXpressCompression supported: 0 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: kZlibCompression supported: 1 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: kSnappyCompression supported: 1 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: kLZ4Compression supported: 1 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: kBZip2Compression supported: 0 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.merge_operator: 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_filter: None 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T06:24:16.983 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a2b05119c0) 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: cache_index_and_filter_blocks: 1 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: pin_top_level_index_and_filter: 1 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_type: 0 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: data_block_index_type: 0 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_shortening: 1 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: checksum: 4 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: no_block_cache: 0 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache: 0x55a2b0515090 2024-02-10T06:24:16.984 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_name: BinnedLRUCache 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_options: 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: capacity : 536870912 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: num_shard_bits : 4 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: strict_capacity_limit : 0 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: high_pri_pool_ratio: 0.000 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_compressed: (nil) 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: persistent_cache: (nil) 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_size: 4096 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_size_deviation: 10 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_restart_interval: 16 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_block_restart_interval: 1 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: metadata_block_size: 4096 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: partition_filters: 0 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: use_delta_encoding: 1 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: filter_policy: bloomfilter 2024-02-10T06:24:16.985 INFO:journalctl@ceph.mon.b.smithi150.stdout: whole_key_filtering: 1 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout: verify_compression: 0 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout: read_amp_bytes_per_bit: 0 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout: format_version: 5 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout: enable_index_compression: 1 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_align: 0 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout: max_auto_readahead_size: 262144 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout: prepopulate_block_cache: 0 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout: initial_auto_readahead_size: 8192 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compression: NoCompression 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T06:24:16.986 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.num_levels: 7 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T06:24:16.987 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T06:24:16.988 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T06:24:16.989 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.table_properties_collectors: 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.inplace_update_support: 0 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T06:24:16.990 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.bloom_locality: 0 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.max_successive_merges: 0 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.ttl: 2592000 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.enable_blob_files: false 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.min_blob_size: 0 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T06:24:16.991 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 54.sst 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 56, last_sequence is 28867, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8e00fb6d-d07f-4008-a352-3511f38022a2 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546256878454, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-02-10T06:24:16.992 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546256885866, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 1649139, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 28872, "largest_seqno": 30477, "table_properties": {"data_size": 1642905, "index_size": 3619, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1605, "raw_key_size": 14915, "raw_average_key_size": 24, "raw_value_size": 1629969, "raw_average_value_size": 2624, "num_data_blocks": 164, "num_entries": 621, "num_filter_entries": 621, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707546256, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8e00fb6d-d07f-4008-a352-3511f38022a2", "db_session_id": "8K94NEGMROMXBOCV6JX1", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546256885980, "job": 1, "event": "recovery_finished"} 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a2b0606000 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: DB pointer 0x55a2b05f0000 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** DB Stats ** 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T06:24:16.993 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: L0 1/0 1.57 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 501.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: L6 1/0 8.97 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: Sum 2/0 10.55 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 501.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 501.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-10T06:24:16.994 INFO:journalctl@ceph.mon.b.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 501.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative compaction: 0.00 GB write, 159.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval compaction: 0.00 GB write, 159.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: Block cache BinnedLRUCache@0x55a2b0515090#2 capacity: 512.00 MB usage: 5.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-02-10T06:24:16.995 INFO:journalctl@ceph.mon.b.smithi150.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: starting mon.b rank 2 at public addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] at bind addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: mon.b@-1(???) e3 preinit fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: mon.b@-1(???).mds e1 new map 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: mon.b@-1(???).mds e1 print_map 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout: e1 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout: legacy client fscid: -1 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout: No filesystems configured 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: mon.b@-1(???).osd e110 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:24:16.996 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:24:16.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:24:16.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-02-10T06:24:16.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 bash[68037]: ec59077c047e2307507159003ccdb94a9ba436e549d632b318e582e84428ac72 2024-02-10T06:24:16.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 systemd[1]: Started Ceph mon.b for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:24:16.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:16.997 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:16 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:17.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:16 smithi139 ceph-mon[24727]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:17.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:16 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:17.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:16 smithi139 ceph-mon[29453]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:17.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:16 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:18.111 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:18.111 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: mon.b calling monitor election 2024-02-10T06:24:18.111 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: mon.a calling monitor election 2024-02-10T06:24:18.111 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T06:24:18.111 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T06:24:18.111 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: fsmap 2024-02-10T06:24:18.111 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:24:18.112 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: mgrmap e42: y(active, since 2m), standbys: x 2024-02-10T06:24:18.112 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: overall HEALTH_OK 2024-02-10T06:24:18.112 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:18.112 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:18.112 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:17 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: mon.b calling monitor election 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: mon.a calling monitor election 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: fsmap 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: mgrmap e42: y(active, since 2m), standbys: x 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: overall HEALTH_OK 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:18.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: mon.b calling monitor election 2024-02-10T06:24:18.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: mon.a calling monitor election 2024-02-10T06:24:18.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T06:24:18.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} 2024-02-10T06:24:18.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: fsmap 2024-02-10T06:24:18.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:24:18.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: mgrmap e42: y(active, since 2m), standbys: x 2024-02-10T06:24:18.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: overall HEALTH_OK 2024-02-10T06:24:18.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:18.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:18.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:17 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:19.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:24:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:24:18] "GET /metrics HTTP/1.1" 200 37930 "" "Prometheus/2.43.0" 2024-02-10T06:24:20.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:20 smithi150 ceph-mon[68055]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:20.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:20 smithi139 ceph-mon[24727]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:20.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:20 smithi139 ceph-mon[29453]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:22.179 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:24:21 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:24:21.894+0000 7f9f98f91700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-10T06:24:22.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:22 smithi139 ceph-mon[29453]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:22.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:22 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:22.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:22 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:22.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:22 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:22.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:22 smithi139 ceph-mon[24727]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:22.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:22 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:22.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:22 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:22.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:22 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:22 smithi150 ceph-mon[68055]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:22.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:22 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:22.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:22 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:22.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:22 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:23.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:23 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:23.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:23 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:23.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:23 smithi150 ceph-mon[68055]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:23.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:23 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:23.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:23 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:23.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:23 smithi139 ceph-mon[24727]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:23.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:23 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:23.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:23 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:23.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:23 smithi139 ceph-mon[29453]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.589 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.591 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:24:26.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:24:26.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:24:26.596 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:24:26.596 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:24:26.596 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:24:26.596 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:24:26.596 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:24:26.596 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:24:26.596 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:24:26.596 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.596 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.596 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:24:26.598 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:24:26.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:24:26.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:24:26.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:24:26.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:24:26.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:24:26.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:24:26.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:24:26.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:26.600 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:26.601 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:26.601 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.601 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:26.601 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:26.601 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:26.601 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.603 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.603 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.603 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.603 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.603 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:26.604 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:26.605 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:26 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:26.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:26.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:26.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:26.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:26.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:24:26.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:24:26.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:24:26.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:24:26.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:24:26.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:24:26.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:24:26.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:24:26.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:24:26.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:24:26.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:24:26.631 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:26.632 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:26.632 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:26.632 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:26.632 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:26.632 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:26.632 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:26.632 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:26 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:27.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Detected new or changed devices on smithi150 2024-02-10T06:24:27.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all crash 2024-02-10T06:24:27.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all mds 2024-02-10T06:24:27.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:24:27.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:24:27.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:24:27.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all nfs 2024-02-10T06:24:27.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:24:27.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all node-exporter 2024-02-10T06:24:27.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all prometheus 2024-02-10T06:24:27.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all alertmanager 2024-02-10T06:24:27.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all grafana 2024-02-10T06:24:27.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all loki 2024-02-10T06:24:27.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all promtail 2024-02-10T06:24:27.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Finalizing container_image settings 2024-02-10T06:24:27.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: Upgrade: Complete! 2024-02-10T06:24:27.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Detected new or changed devices on smithi150 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all crash 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all mds 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all nfs 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all node-exporter 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all prometheus 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all alertmanager 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all grafana 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all loki 2024-02-10T06:24:27.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all promtail 2024-02-10T06:24:27.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Finalizing container_image settings 2024-02-10T06:24:27.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: Upgrade: Complete! 2024-02-10T06:24:27.590 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:27 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Detected new or changed devices on smithi150 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all crash 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mds 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all nfs 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all node-exporter 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all prometheus 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all alertmanager 2024-02-10T06:24:27.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all grafana 2024-02-10T06:24:27.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all loki 2024-02-10T06:24:27.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all promtail 2024-02-10T06:24:27.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Finalizing container_image settings 2024-02-10T06:24:27.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: Upgrade: Complete! 2024-02-10T06:24:27.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:27 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:28 smithi139 ceph-mon[24727]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:28.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:28 smithi139 ceph-mon[29453]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:28.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:28 smithi150 ceph-mon[68055]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:29.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:24:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:24:28] "GET /metrics HTTP/1.1" 200 37869 "" "Prometheus/2.43.0" 2024-02-10T06:24:30.431 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:30 smithi150 ceph-mon[68055]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:30.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:30 smithi139 ceph-mon[24727]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:30.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:30 smithi139 ceph-mon[29453]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:32.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:32 smithi139 ceph-mon[24727]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:32.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:32 smithi139 ceph-mon[29453]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:32.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:32 smithi150 ceph-mon[68055]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:34.550 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:34 smithi139 ceph-mon[24727]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:34.550 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:34 smithi139 ceph-mon[29453]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:34.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:34 smithi150 ceph-mon[68055]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:36.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:36 smithi139 ceph-mon[24727]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:36.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:36 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:36 smithi139 ceph-mon[29453]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:36.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:36 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:36.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:36 smithi150 ceph-mon[68055]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:36.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:36 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:37.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:37 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:37.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:37 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:37.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:37 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:38 smithi139 ceph-mon[24727]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:38 smithi139 ceph-mon[29453]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:38 smithi150 ceph-mon[68055]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:39.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:24:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:24:38] "GET /metrics HTTP/1.1" 200 37942 "" "Prometheus/2.43.0" 2024-02-10T06:24:40.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:40 smithi139 ceph-mon[24727]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:40.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:40 smithi139 ceph-mon[29453]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:40.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:40 smithi150 ceph-mon[68055]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:42.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:42 smithi139 ceph-mon[29453]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:42.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:42 smithi139 ceph-mon[24727]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:42.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:42 smithi150 ceph-mon[68055]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:43.065 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:24:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:44 smithi139 ceph-mon[24727]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:44 smithi139 ceph-mon[24727]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:44.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:44 smithi139 ceph-mon[29453]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:44.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:44 smithi139 ceph-mon[29453]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:44.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:44 smithi150 ceph-mon[68055]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:44.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:44 smithi150 ceph-mon[68055]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:45.102 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:24:45.102 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (103s) 83s ago 23m 16.8M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:24:45.102 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (87s) 23s ago 23m 84.2M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (111s) 83s ago 22m 50.3M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (2m) 23s ago 28m 450M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (12m) 83s ago 29m 504M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (29m) 83s ago 30m 76.5M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (28s) 23s ago 28m 18.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (28m) 83s ago 28m 55.2M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (2m) 83s ago 24m 14.9M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (2m) 23s ago 24m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (27m) 83s ago 27m 54.5M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (27m) 83s ago 27m 55.1M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (27m) 83s ago 27m 53.3M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:24:45.103 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (26m) 83s ago 26m 53.5M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:24:45.104 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (26m) 23s ago 26m 55.9M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:24:45.104 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (26m) 23s ago 26m 55.5M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:24:45.104 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (25m) 23s ago 25m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:24:45.104 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (25m) 23s ago 25m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:24:45.104 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (2m) 23s ago 24m 53.2M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:24:45.104 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (23m) 83s ago 23m 75.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:24:45.104 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (23m) 23s ago 23m 75.5M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:24:45.826 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 2'"'"'' 2024-02-10T06:24:46.470 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:46 smithi139 ceph-mon[24727]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:46.470 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:46 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:46.470 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:46 smithi139 ceph-mon[24727]: from='client.25687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:46.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:46 smithi139 ceph-mon[29453]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:46.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:46 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:46.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:46 smithi139 ceph-mon[29453]: from='client.25687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:46 smithi150 ceph-mon[68055]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:46 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:46 smithi150 ceph-mon[68055]: from='client.25687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:47.959 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:24:48.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:48 smithi139 ceph-mon[24727]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:48.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:48 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/2605831512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:48 smithi139 ceph-mon[29453]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:48 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/2605831512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:48 smithi150 ceph-mon[68055]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:48.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:48 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/2605831512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:24:48.647 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade status' 2024-02-10T06:24:48.955 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:24:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:24:48] "GET /metrics HTTP/1.1" 200 37942 "" "Prometheus/2.43.0" 2024-02-10T06:24:50.442 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:50 smithi150 ceph-mon[68055]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:50.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:50 smithi139 ceph-mon[24727]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:50.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:50 smithi139 ceph-mon[29453]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:50.631 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:24:50.632 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": null, 2024-02-10T06:24:50.632 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": false, 2024-02-10T06:24:50.632 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "", 2024-02-10T06:24:50.632 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:24:50.632 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": null, 2024-02-10T06:24:50.632 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "", 2024-02-10T06:24:50.632 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:24:50.632 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:24:51.259 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T06:24:51.561 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:51 smithi139 ceph-mon[24727]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:51.561 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:51 smithi139 ceph-mon[24727]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:51.562 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:51 smithi139 ceph-mon[29453]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:51.562 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:51 smithi139 ceph-mon[29453]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:51.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:51 smithi150 ceph-mon[68055]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:51.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:51 smithi150 ceph-mon[68055]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:52.543 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:52 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:52.543 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:52 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:52.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:52 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:53.301 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_OK 2024-02-10T06:24:53.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:53 smithi139 ceph-mon[29453]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:53.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:53 smithi139 ceph-mon[24727]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:53.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:53 smithi150 ceph-mon[68055]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:53.883 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '"'"'{print $2}'"'"')' 2024-02-10T06:24:54.443 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:54 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/1525375671' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:24:54.443 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:54 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/1525375671' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:24:54.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:54 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/1525375671' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:24:55.436 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:55 smithi139 ceph-mon[29453]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:55.436 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:55 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:55.436 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:55 smithi139 ceph-mon[24727]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:55.436 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:55 smithi139 ceph-mon[24727]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:55.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:55 smithi150 ceph-mon[68055]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:55.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:55 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:56.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:56 smithi139 ceph-mon[24727]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:56.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:56 smithi139 ceph-mon[29453]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:56.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:56 smithi150 ceph-mon[68055]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:57.441 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:57 smithi139 ceph-mon[24727]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mon", "hosts": "smithi139", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:57.441 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:57 smithi139 ceph-mon[24727]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:57.441 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:57 smithi139 ceph-mon[29453]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mon", "hosts": "smithi139", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:57.441 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:57 smithi139 ceph-mon[29453]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:57 smithi150 ceph-mon[68055]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mon", "hosts": "smithi139", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:24:57.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:57 smithi150 ceph-mon[68055]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:58.369 INFO:teuthology.orchestra.run.smithi139.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:58.998 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:24:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:24:58] "GET /metrics HTTP/1.1" 200 37939 "" "Prometheus/2.43.0" 2024-02-10T06:24:59.020 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T06:24:59.593 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[29453]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[29453]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[29453]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[24727]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:59.594 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:59.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[24727]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:59.595 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:24:59 smithi139 ceph-mon[24727]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:59.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:59 smithi150 ceph-mon[68055]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:59.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:59 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:59.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:59 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:59.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:59 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:59.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:59 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:59.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:59 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:24:59.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:59 smithi150 ceph-mon[68055]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:24:59.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:24:59 smithi150 ceph-mon[68055]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:01.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[24727]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[29453]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:25:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:25:01.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:01 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-10T06:25:01.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:01 smithi150 ceph-mon[68055]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:01.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:01 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:25:01.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:01 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:01.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:01 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:01.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:01 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:25:01.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:01 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:25:01.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:01 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-10T06:25:02.790 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:02 smithi139 ceph-mon[24727]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:25:02.790 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:02 smithi139 ceph-mon[24727]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:25:02.790 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:02 smithi139 ceph-mon[24727]: Upgrade: Setting container_image for all mgr 2024-02-10T06:25:02.790 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:02 smithi139 ceph-mon[24727]: Upgrade: It appears safe to stop mon.a 2024-02-10T06:25:02.790 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:02 smithi139 ceph-mon[29453]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:25:02.791 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:02 smithi139 ceph-mon[29453]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:25:02.791 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:02 smithi139 ceph-mon[29453]: Upgrade: Setting container_image for all mgr 2024-02-10T06:25:02.791 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:02 smithi139 ceph-mon[29453]: Upgrade: It appears safe to stop mon.a 2024-02-10T06:25:02.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:02 smithi150 ceph-mon[68055]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:25:02.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:02 smithi150 ceph-mon[68055]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:25:02.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:02 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mgr 2024-02-10T06:25:02.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:02 smithi150 ceph-mon[68055]: Upgrade: It appears safe to stop mon.a 2024-02-10T06:25:03.060 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:25:03.754 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:25:03.754 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (2m) 101s ago 24m 16.8M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:25:03.754 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (106s) 42s ago 23m 84.2M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:25:03.754 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (2m) 101s ago 23m 50.3M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:25:03.754 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (3m) 42s ago 28m 450M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (13m) 101s ago 30m 504M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (30m) 101s ago 30m 76.5M 2048M 17.2.0 e1d6a67b021e 2a614b8dfab7 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (47s) 42s ago 28m 18.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (29m) 101s ago 29m 55.2M 2048M 17.2.0 e1d6a67b021e 0ed77770af21 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (2m) 101s ago 24m 14.9M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (2m) 42s ago 24m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (28m) 101s ago 28m 54.5M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (27m) 101s ago 27m 55.1M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (27m) 101s ago 27m 53.3M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (27m) 101s ago 27m 53.5M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (26m) 42s ago 26m 55.9M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (26m) 42s ago 26m 55.5M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (26m) 42s ago 26m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (25m) 42s ago 25m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:25:03.755 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (2m) 42s ago 24m 53.2M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:25:03.756 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (23m) 101s ago 23m 75.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:25:03.756 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (23m) 42s ago 23m 75.5M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:25:04.166 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:25:04.166 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:25:04.166 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-10T06:25:04.166 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:25:04.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": {}, 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:25:04.167 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:25:04.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[24727]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[24727]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[24727]: Upgrade: Updating mon.a 2024-02-10T06:25:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[24727]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:25:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[24727]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[24727]: Deploying daemon mon.a on smithi139 2024-02-10T06:25:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[24727]: from='client.? 172.21.15.139:0/253115940' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:04.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[29453]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:04.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[29453]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:04.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[29453]: Upgrade: Updating mon.a 2024-02-10T06:25:04.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:25:04.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:04.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:04.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:04.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[29453]: Deploying daemon mon.a on smithi139 2024-02-10T06:25:04.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:04 smithi139 ceph-mon[29453]: from='client.? 172.21.15.139:0/253115940' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:04.494 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:25:04.494 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:25:04.494 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:25:04.494 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi139", 2024-02-10T06:25:04.494 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:25:04.494 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "0/2 daemons upgraded", 2024-02-10T06:25:04.494 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading mon daemons", 2024-02-10T06:25:04.495 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:25:04.495 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:25:04.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:04 smithi150 ceph-mon[68055]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:04.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:04 smithi150 ceph-mon[68055]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:04.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:04 smithi150 ceph-mon[68055]: Upgrade: Updating mon.a 2024-02-10T06:25:04.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:04 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='mgr.y' 2024-02-10T06:25:04.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:04 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:04.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:04 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:04.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:04 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:04.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:04 smithi150 ceph-mon[68055]: Deploying daemon mon.a on smithi139 2024-02-10T06:25:04.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:04 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/253115940' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:05.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:05 smithi139 systemd[1]: Stopping Ceph mon.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:25:05.433 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:05 smithi139 ceph-mon[29453]: from='client.15825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:05.434 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:05 smithi139 ceph-mon[29453]: from='client.15831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:05.434 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:05 smithi139 ceph-mon[24727]: from='client.15825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:05.434 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:05 smithi139 ceph-mon[24727]: from='client.15831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:05.434 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:05 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a[24723]: 2024-02-10T06:25:05.269+0000 7fcdba227700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:25:05.434 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:05 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a[24723]: 2024-02-10T06:25:05.269+0000 7fcdba227700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-10T06:25:05.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:05 smithi150 ceph-mon[68055]: from='client.15825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:05.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:05 smithi150 ceph-mon[68055]: from='client.15831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:05.728 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:05 smithi139 podman[85314]: 2024-02-10 06:25:05.433271311 +0000 UTC m=+0.286478140 container died 2a614b8dfab775853ea8dab65e69d211335c9a1ad1656141abd37faa8e909f06 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, com.redhat.component=centos-stream-container, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, version=8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , ceph=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_BRANCH=HEAD) 2024-02-10T06:25:06.078 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:05 smithi139 podman[85314]: 2024-02-10 06:25:05.818992153 +0000 UTC m=+0.672198983 container cleanup 2a614b8dfab775853ea8dab65e69d211335c9a1ad1656141abd37faa8e909f06 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, com.redhat.component=centos-stream-container, RELEASE=HEAD, maintainer=Guillaume Abrioux , distribution-scope=public, name=centos-stream, GIT_CLEAN=True, ceph=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_BRANCH=HEAD, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, release=754, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream) 2024-02-10T06:25:06.079 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:05 smithi139 bash[85314]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a 2024-02-10T06:25:06.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 podman[85328]: 2024-02-10 06:25:06.078357102 +0000 UTC m=+0.637200136 container remove 2a614b8dfab775853ea8dab65e69d211335c9a1ad1656141abd37faa8e909f06 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, ceph=True, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, name=centos-stream) 2024-02-10T06:25:06.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-10T06:25:06.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 systemd[1]: Stopped Ceph mon.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:25:06.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.a.service: Consumed 26.105s CPU time. 2024-02-10T06:25:06.815 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 systemd[1]: Starting Ceph mon.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:25:06.815 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 podman[85461]: 2024-02-10 06:25:06.569339922 +0000 UTC m=+0.018137741 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:25:06.815 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 podman[85461]: 2024-02-10 06:25:06.688109741 +0000 UTC m=+0.136907587 container create ee03df6a685e43fe9b26357bff470e61f785fef52f608e587f6ac23189a18aa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True) 2024-02-10T06:25:07.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 podman[85461]: 2024-02-10 06:25:06.924168034 +0000 UTC m=+0.372965853 container init ee03df6a685e43fe9b26357bff470e61f785fef52f608e587f6ac23189a18aa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-10T06:25:07.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 podman[85461]: 2024-02-10 06:25:06.927456514 +0000 UTC m=+0.376254328 container start ee03df6a685e43fe9b26357bff470e61f785fef52f608e587f6ac23189a18aa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:25:07.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T06:25:07.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T06:25:07.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: pidfile_write: ignore empty --pid-file 2024-02-10T06:25:07.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: load: jerasure load: lrc 2024-02-10T06:25:07.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: RocksDB version: 7.9.2 2024-02-10T06:25:07.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Git sha 0 2024-02-10T06:25:07.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: DB SUMMARY 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: DB Session ID: 7MNEBSON0YB8N8REG04I 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: CURRENT file: CURRENT 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: MANIFEST file: MANIFEST-000015 size: 3080 Bytes 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000063.sst 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000061.log size: 142453 ; 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.error_if_exists: 0 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.create_if_missing: 0 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.paranoid_checks: 1 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.env: 0x55782d13dae0 2024-02-10T06:25:07.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T06:25:07.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.info_log: 0x55782da358c0 2024-02-10T06:25:07.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T06:25:07.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.statistics: (nil) 2024-02-10T06:25:07.090 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.use_fsync: 0 2024-02-10T06:25:07.090 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_log_file_size: 0 2024-02-10T06:25:07.090 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T06:25:07.090 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T06:25:07.090 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.allow_fallocate: 1 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.use_direct_reads: 0 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.db_log_dir: 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.wal_dir: 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T06:25:07.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.write_buffer_manager: 0x55782da08dc0 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.unordered_write: 0 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T06:25:07.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.row_cache: None 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.wal_filter: None 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.two_write_queues: 0 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.wal_compression: 0 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.atomic_flush: 0 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.log_readahead_size: 0 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T06:25:07.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_background_jobs: 2 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_background_compactions: -1 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_subcompactions: 1 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T06:25:07.094 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_open_files: -1 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_background_flushes: -1 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Compression algorithms supported: 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: kZSTD supported: 0 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: kXpressCompression supported: 0 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: kZlibCompression supported: 1 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: kSnappyCompression supported: 1 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: kLZ4Compression supported: 1 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: kBZip2Compression supported: 0 2024-02-10T06:25:07.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.merge_operator: 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_filter: None 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55782da359c0) 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout: cache_index_and_filter_blocks: 1 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T06:25:07.096 INFO:journalctl@ceph.mon.a.smithi139.stdout: pin_top_level_index_and_filter: 1 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: index_type: 0 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: data_block_index_type: 0 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: index_shortening: 1 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: checksum: 4 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: no_block_cache: 0 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_cache: 0x55782da39090 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_cache_name: BinnedLRUCache 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_cache_options: 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: capacity : 536870912 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: num_shard_bits : 4 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: strict_capacity_limit : 0 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: high_pri_pool_ratio: 0.000 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_cache_compressed: (nil) 2024-02-10T06:25:07.097 INFO:journalctl@ceph.mon.a.smithi139.stdout: persistent_cache: (nil) 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_size: 4096 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_size_deviation: 10 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_restart_interval: 16 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: index_block_restart_interval: 1 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: metadata_block_size: 4096 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: partition_filters: 0 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: use_delta_encoding: 1 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: filter_policy: bloomfilter 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: whole_key_filtering: 1 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: verify_compression: 0 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: read_amp_bytes_per_bit: 0 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: format_version: 5 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: enable_index_compression: 1 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: block_align: 0 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: max_auto_readahead_size: 262144 2024-02-10T06:25:07.098 INFO:journalctl@ceph.mon.a.smithi139.stdout: prepopulate_block_cache: 0 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout: initial_auto_readahead_size: 8192 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compression: NoCompression 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.num_levels: 7 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T06:25:07.099 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T06:25:07.100 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T06:25:07.101 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T06:25:07.102 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.table_properties_collectors: 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.inplace_update_support: 0 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.bloom_locality: 0 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.max_successive_merges: 0 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.ttl: 2592000 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T06:25:07.103 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.enable_blob_files: false 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.min_blob_size: 0 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 63.sst 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 65, last_sequence is 29287, log_number is 61,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 61 2024-02-10T06:25:07.104 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: eacfdd00-41a0-4e2a-befa-61a7562c02be 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546306957343, "job": 1, "event": "recovery_started", "wal_files": [61]} 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #61 mode 2 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546306958556, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 66, "file_size": 140363, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 29288, "largest_seqno": 29458, "table_properties": {"data_size": 138710, "index_size": 385, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 261, "raw_key_size": 2301, "raw_average_key_size": 25, "raw_value_size": 136763, "raw_average_value_size": 1502, "num_data_blocks": 18, "num_entries": 91, "num_filter_entries": 91, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707546306, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "eacfdd00-41a0-4e2a-befa-61a7562c02be", "db_session_id": "7MNEBSON0YB8N8REG04I", "orig_file_number": 66, "seqno_to_time_mapping": "N/A"}} 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546306958637, "job": 1, "event": "recovery_finished"} 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/version_set.cc:5047] Creating manifest 68 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000061.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55782db2a000 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: DB pointer 0x55782db14000 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout: ** DB Stats ** 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:25:07.105 INFO:journalctl@ceph.mon.a.smithi139.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: L0 1/0 137.07 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: L6 1/0 9.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: Sum 2/0 9.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T06:25:07.106 INFO:journalctl@ceph.mon.a.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: Cumulative compaction: 0.00 GB write, 40.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:25:07.107 INFO:journalctl@ceph.mon.a.smithi139.stdout: Interval compaction: 0.00 GB write, 40.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:25:07.108 INFO:journalctl@ceph.mon.a.smithi139.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout: Block cache BinnedLRUCache@0x55782da39090#2 capacity: 512.00 MB usage: 0.80 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.31 KB,5.96046e-05%) IndexBlock(1,0.48 KB,9.23872e-05%) Misc(1,0.00 KB,0%) 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: starting mon.a rank 0 at public addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] at bind addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: mon.a@-1(???) e3 preinit fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: mon.a@-1(???).mds e1 new map 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: mon.a@-1(???).mds e1 print_map 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout: e1 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout: legacy client fscid: -1 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout: 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout: No filesystems configured 2024-02-10T06:25:07.109 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: mon.a@-1(???).osd e110 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-10T06:25:07.110 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: mon.a@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:25:07.110 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: mon.a@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:25:07.110 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: mon.a@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:25:07.110 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:06 smithi139 ceph-mon[85480]: mon.a@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-02-10T06:25:07.110 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:07 smithi139 bash[85461]: ee03df6a685e43fe9b26357bff470e61f785fef52f608e587f6ac23189a18aa0 2024-02-10T06:25:07.110 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:07 smithi139 systemd[1]: Started Ceph mon.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:25:08.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ignoring --setuser ceph since I am not root 2024-02-10T06:25:08.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ignoring --setgroup ceph since I am not root 2024-02-10T06:25:08.837 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mgr[63877]: -- 172.21.15.139:0/1616853979 <== mon.0 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560198ae01a0 con 0x560198ab7000 2024-02-10T06:25:08.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[29453]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:08.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[29453]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:08.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[29453]: mon.a calling monitor election 2024-02-10T06:25:08.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[29453]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T06:25:08.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[29453]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:25:08.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[29453]: fsmap 2024-02-10T06:25:08.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[29453]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:25:08.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[29453]: mgrmap e42: y(active, since 3m), standbys: x 2024-02-10T06:25:08.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[29453]: from='mgr.25240 ' entity='' 2024-02-10T06:25:08.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[29453]: mgrmap e43: y(active, since 3m), standbys: x 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[29453]: overall HEALTH_OK 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[85480]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[85480]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[85480]: mon.a calling monitor election 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[85480]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[85480]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[85480]: fsmap 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[85480]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[85480]: mgrmap e42: y(active, since 3m), standbys: x 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[85480]: from='mgr.25240 ' entity='' 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[85480]: mgrmap e43: y(active, since 3m), standbys: x 2024-02-10T06:25:08.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-mon[85480]: overall HEALTH_OK 2024-02-10T06:25:08.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: ignoring --setuser ceph since I am not root 2024-02-10T06:25:08.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: ignoring --setgroup ceph since I am not root 2024-02-10T06:25:08.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mgr[61242]: -- 172.21.15.150:0/1409287109 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560be6aea1a0 con 0x560be6ac1000 2024-02-10T06:25:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mon[68055]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mon[68055]: from='mgr.25240 172.21.15.139:0/3713564553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mon[68055]: mon.a calling monitor election 2024-02-10T06:25:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mon[68055]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T06:25:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mon[68055]: monmap e3: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:25:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mon[68055]: fsmap 2024-02-10T06:25:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mon[68055]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:25:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mon[68055]: mgrmap e42: y(active, since 3m), standbys: x 2024-02-10T06:25:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mon[68055]: from='mgr.25240 ' entity='' 2024-02-10T06:25:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mon[68055]: mgrmap e43: y(active, since 3m), standbys: x 2024-02-10T06:25:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-mon[68055]: overall HEALTH_OK 2024-02-10T06:25:09.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:08.982+0000 7ff66f775200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T06:25:09.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:09.078+0000 7ff66f775200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T06:25:09.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:08.987+0000 7f07d64b9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T06:25:09.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:09.082+0000 7f07d64b9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T06:25:10.052 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:09.718+0000 7ff66f775200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T06:25:10.053 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:09.811+0000 7ff66f775200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T06:25:10.053 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:09.954+0000 7ff66f775200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T06:25:10.053 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:10.051+0000 7ff66f775200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T06:25:10.056 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:09.723+0000 7f07d64b9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T06:25:10.057 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:09.815+0000 7f07d64b9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T06:25:10.057 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:09.958+0000 7f07d64b9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T06:25:10.057 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:10.056+0000 7f07d64b9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T06:25:10.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:10.136+0000 7ff66f775200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T06:25:10.374 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:10.139+0000 7f07d64b9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T06:25:10.694 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:10 smithi139 ceph-mon[29453]: mgrmap e44: y(active, since 3m), standbys: x 2024-02-10T06:25:10.695 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:10 smithi139 ceph-mon[85480]: mgrmap e44: y(active, since 3m), standbys: x 2024-02-10T06:25:10.695 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:10.402+0000 7ff66f775200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T06:25:10.695 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:10.511+0000 7ff66f775200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T06:25:10.695 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:10.598+0000 7ff66f775200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T06:25:10.695 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:10.694+0000 7ff66f775200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T06:25:10.696 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:10 smithi150 ceph-mon[68055]: mgrmap e44: y(active, since 3m), standbys: x 2024-02-10T06:25:10.696 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:10.404+0000 7f07d64b9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T06:25:10.697 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:10.513+0000 7f07d64b9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T06:25:10.697 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:10.600+0000 7f07d64b9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T06:25:10.697 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:10.695+0000 7f07d64b9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T06:25:11.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:10 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:10.872+0000 7ff66f775200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T06:25:11.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:10.873+0000 7f07d64b9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T06:25:12.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:11 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:11.851+0000 7f07d64b9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T06:25:12.124 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:11 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:11.984+0000 7f07d64b9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T06:25:12.320 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:11 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:11.844+0000 7ff66f775200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T06:25:12.321 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:11 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:11.979+0000 7ff66f775200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T06:25:12.321 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:12.319+0000 7ff66f775200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T06:25:12.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:12.412+0000 7ff66f775200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T06:25:12.618 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:12 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:12.325+0000 7f07d64b9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T06:25:12.618 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:12 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:12.417+0000 7f07d64b9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T06:25:12.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:12 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:12.617+0000 7f07d64b9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:25:12.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:12 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:12.806+0000 7f07d64b9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T06:25:13.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:12.612+0000 7ff66f775200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:25:13.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:12 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:12.806+0000 7ff66f775200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T06:25:13.358 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:13.100+0000 7ff66f775200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:25:13.365 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:13.102+0000 7f07d64b9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T06:25:13.614 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:13.372+0000 7ff66f775200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T06:25:13.615 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:13.613+0000 7ff66f775200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T06:25:13.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:13.364+0000 7f07d64b9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T06:25:13.624 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:13.607+0000 7f07d64b9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T06:25:14.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:14 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:14.432+0000 7ff66f775200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T06:25:14.874 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:14.428+0000 7f07d64b9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T06:25:15.335 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:15.005+0000 7f07d64b9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T06:25:15.335 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:15.108+0000 7f07d64b9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T06:25:15.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:15.016+0000 7ff66f775200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T06:25:15.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:15.118+0000 7ff66f775200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T06:25:15.593 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:15.333+0000 7f07d64b9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T06:25:15.593 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: [10/Feb/2024:06:25:15] ENGINE Bus STARTING 2024-02-10T06:25:15.593 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: CherryPy Checker: 2024-02-10T06:25:15.593 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: The Application mounted at '' has an empty config. 2024-02-10T06:25:15.593 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: 2024-02-10T06:25:15.593 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: [10/Feb/2024:06:25:15] ENGINE Serving on http://:::9283 2024-02-10T06:25:15.593 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x[61238]: [10/Feb/2024:06:25:15] ENGINE Bus STARTED 2024-02-10T06:25:15.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: Standby manager daemon x restarted 2024-02-10T06:25:15.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: Standby manager daemon x started 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: Active manager daemon y restarted 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: Activating manager daemon y 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: mgrmap e45: y(active, starting, since 0.00691025s), standbys: x 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:25:15.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:25:15.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:25:15.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:25:15.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:25:15.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:25:15.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:25:15.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:25:15.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:25:15.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:25:15.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:25:15.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:15 smithi150 ceph-mon[68055]: Manager daemon y is now available 2024-02-10T06:25:15.596 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:15.341+0000 7ff66f775200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T06:25:15.596 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:25:15] ENGINE Bus STARTING 2024-02-10T06:25:15.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c[29449]: 2024-02-10T06:25:15.359+0000 7f8ac33f7700 -1 mon.c@1(peon).osd e111 update_from_paxos full map CRC mismatch, resetting to canonical 2024-02-10T06:25:15.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: Standby manager daemon x restarted 2024-02-10T06:25:15.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: Standby manager daemon x started 2024-02-10T06:25:15.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:25:15.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:25:15.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:25:15.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:25:15.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: Active manager daemon y restarted 2024-02-10T06:25:15.597 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: Activating manager daemon y 2024-02-10T06:25:15.598 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:25:15.598 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: mgrmap e45: y(active, starting, since 0.00691025s), standbys: x 2024-02-10T06:25:15.598 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:25:15.598 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:25:15.598 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:25:15.598 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:25:15.598 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:25:15.598 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:25:15.598 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:25:15.598 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:25:15.598 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[29453]: Manager daemon y is now available 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: Standby manager daemon x restarted 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: Standby manager daemon x started 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:25:15.599 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.? 172.21.15.150:0/3709073809' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: Active manager daemon y restarted 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: Activating manager daemon y 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: mgrmap e45: y(active, starting, since 0.00691025s), standbys: x 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:25:15.600 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:25:15.601 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:25:15.601 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:25:15.601 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:25:15.601 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:25:15.601 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:25:15.601 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:25:15.601 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-mon[85480]: Manager daemon y is now available 2024-02-10T06:25:16.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: CherryPy Checker: 2024-02-10T06:25:16.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: The Application mounted at '' has an empty config. 2024-02-10T06:25:16.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:16.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:25:15] ENGINE Serving on http://:::9283 2024-02-10T06:25:16.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:15 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:25:15] ENGINE Bus STARTED 2024-02-10T06:25:16.627 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[85480]: failed to encode map e111 with expected crc 2024-02-10T06:25:16.627 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[85480]: failed to encode map e111 with expected crc 2024-02-10T06:25:16.627 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:16.627 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:16.627 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:16.628 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:25:16.628 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:25:16.628 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[85480]: failed to encode map e111 with expected crc 2024-02-10T06:25:16.628 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:25:16.628 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:25:16.628 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[85480]: mgrmap e46: y(active, since 1.00949s), standbys: x 2024-02-10T06:25:16.628 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:16.366+0000 7ff625973700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-02-10T06:25:16.628 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[29453]: failed to encode map e111 with expected crc 2024-02-10T06:25:16.629 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[29453]: failed to encode map e111 with expected crc 2024-02-10T06:25:16.629 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:16.629 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:16.629 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:16.629 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:25:16.629 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:25:16.629 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[29453]: failed to encode map e111 with expected crc 2024-02-10T06:25:16.629 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:25:16.629 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:25:16.629 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:16 smithi139 ceph-mon[29453]: mgrmap e46: y(active, since 1.00949s), standbys: x 2024-02-10T06:25:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:16 smithi150 ceph-mon[68055]: failed to encode map e111 with expected crc 2024-02-10T06:25:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:16 smithi150 ceph-mon[68055]: failed to encode map e111 with expected crc 2024-02-10T06:25:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:25:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:25:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:16 smithi150 ceph-mon[68055]: failed to encode map e111 with expected crc 2024-02-10T06:25:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:25:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T06:25:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:16 smithi150 ceph-mon[68055]: mgrmap e46: y(active, since 1.00949s), standbys: x 2024-02-10T06:25:17.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[29453]: failed to encode map e111 with expected crc 2024-02-10T06:25:17.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[29453]: [10/Feb/2024:06:25:16] ENGINE Bus STARTING 2024-02-10T06:25:17.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[29453]: [10/Feb/2024:06:25:16] ENGINE Serving on http://172.21.15.139:8765 2024-02-10T06:25:17.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[29453]: failed to encode map e111 with expected crc 2024-02-10T06:25:17.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[29453]: [10/Feb/2024:06:25:16] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T06:25:17.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[29453]: [10/Feb/2024:06:25:16] ENGINE Bus STARTED 2024-02-10T06:25:17.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:17.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[29453]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:25:17.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[85480]: failed to encode map e111 with expected crc 2024-02-10T06:25:17.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[85480]: [10/Feb/2024:06:25:16] ENGINE Bus STARTING 2024-02-10T06:25:17.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[85480]: [10/Feb/2024:06:25:16] ENGINE Serving on http://172.21.15.139:8765 2024-02-10T06:25:17.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[85480]: failed to encode map e111 with expected crc 2024-02-10T06:25:17.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[85480]: [10/Feb/2024:06:25:16] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T06:25:17.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[85480]: [10/Feb/2024:06:25:16] ENGINE Bus STARTED 2024-02-10T06:25:17.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:17.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:17 smithi139 ceph-mon[85480]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:25:17.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:17 smithi150 ceph-mon[68055]: failed to encode map e111 with expected crc 2024-02-10T06:25:17.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:17 smithi150 ceph-mon[68055]: [10/Feb/2024:06:25:16] ENGINE Bus STARTING 2024-02-10T06:25:17.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:17 smithi150 ceph-mon[68055]: [10/Feb/2024:06:25:16] ENGINE Serving on http://172.21.15.139:8765 2024-02-10T06:25:17.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:17 smithi150 ceph-mon[68055]: failed to encode map e111 with expected crc 2024-02-10T06:25:17.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:17 smithi150 ceph-mon[68055]: [10/Feb/2024:06:25:16] ENGINE Serving on https://172.21.15.139:7150 2024-02-10T06:25:17.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:17 smithi150 ceph-mon[68055]: [10/Feb/2024:06:25:16] ENGINE Bus STARTED 2024-02-10T06:25:17.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:17 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:17.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:17 smithi150 ceph-mon[68055]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:25:18.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:18 smithi139 ceph-mon[85480]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:25:18.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:18 smithi139 ceph-mon[85480]: mgrmap e47: y(active, since 3s), standbys: x 2024-02-10T06:25:18.677 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:18 smithi139 ceph-mon[29453]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:25:18.677 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:18 smithi139 ceph-mon[29453]: mgrmap e47: y(active, since 3s), standbys: x 2024-02-10T06:25:18.755 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:18 smithi150 ceph-mon[68055]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:25:18.755 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:18 smithi150 ceph-mon[68055]: mgrmap e47: y(active, since 3s), standbys: x 2024-02-10T06:25:19.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:25:18] "GET /metrics HTTP/1.1" 200 35142 "" "Prometheus/2.43.0" 2024-02-10T06:25:20.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:20 smithi150 ceph-mon[68055]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:25:20.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:20 smithi150 ceph-mon[68055]: mgrmap e48: y(active, since 4s), standbys: x 2024-02-10T06:25:20.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:20 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:20.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:20 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:20.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:20 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:20.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:20 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:20.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[85480]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:25:20.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[85480]: mgrmap e48: y(active, since 4s), standbys: x 2024-02-10T06:25:20.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:20.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:20.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:20.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[29453]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:25:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[29453]: mgrmap e48: y(active, since 4s), standbys: x 2024-02-10T06:25:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:20 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:21 smithi139 ceph-mon[85480]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T06:25:22.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:21 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:21 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:21 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:21 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:21 smithi139 ceph-mon[29453]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T06:25:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:22.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:21 smithi150 ceph-mon[68055]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T06:25:24.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:24 smithi139 ceph-mon[85480]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:25:24.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:24 smithi139 ceph-mon[29453]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:25:24.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:24 smithi150 ceph-mon[68055]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T06:25:25.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:25.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:25.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:25.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:25:25.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:25:25.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:25:25.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:25.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:25.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:25.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:25:25.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[85480]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:25:25.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:25 smithi139 ceph-mon[85480]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:25:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:25 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:25 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:25 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:25 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:25:25.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:25 smithi150 ceph-mon[68055]: Updating smithi139:/etc/ceph/ceph.conf 2024-02-10T06:25:25.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:25 smithi150 ceph-mon[68055]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-10T06:25:26.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:26.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:26.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:25:26.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:26 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-10T06:25:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:25:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:25:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:25:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:25:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: Updating smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:25:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: Updating smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:25:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T06:25:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:26.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:26.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:26.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:25:26.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:26 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-10T06:25:27.756 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[85480]: Upgrade: It appears safe to stop mon.c 2024-02-10T06:25:27.756 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:27.756 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:27.756 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:27.756 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:27.756 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:27.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[29453]: Upgrade: It appears safe to stop mon.c 2024-02-10T06:25:27.757 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[29453]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:27.757 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[29453]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:27.757 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:27.757 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:27.757 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:27 smithi139 ceph-mon[29453]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:27.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:27 smithi150 ceph-mon[68055]: Upgrade: It appears safe to stop mon.c 2024-02-10T06:25:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:27 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:27 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:27 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:27 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:27 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:28.746 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:25:28] "GET /metrics HTTP/1.1" 200 35142 "" "Prometheus/2.43.0" 2024-02-10T06:25:28.747 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:28 smithi139 ceph-mon[85480]: Upgrade: Updating mon.c 2024-02-10T06:25:28.747 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:28 smithi139 ceph-mon[85480]: Deploying daemon mon.c on smithi139 2024-02-10T06:25:28.747 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:28 smithi139 ceph-mon[85480]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:25:28.747 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:28 smithi139 ceph-mon[29453]: Upgrade: Updating mon.c 2024-02-10T06:25:28.747 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:28 smithi139 ceph-mon[29453]: Deploying daemon mon.c on smithi139 2024-02-10T06:25:28.747 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:28 smithi139 ceph-mon[29453]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:25:28.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:28 smithi150 ceph-mon[68055]: Upgrade: Updating mon.c 2024-02-10T06:25:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:28 smithi150 ceph-mon[68055]: Deploying daemon mon.c on smithi139 2024-02-10T06:25:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:28 smithi150 ceph-mon[68055]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T06:25:29.312 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:29 smithi139 systemd[1]: Stopping Ceph mon.c for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:25:29.312 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c[29449]: 2024-02-10T06:25:29.298+0000 7f8ac8c02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:25:29.312 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c[29449]: 2024-02-10T06:25:29.298+0000 7f8ac8c02700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-10T06:25:29.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:29 smithi139 podman[87688]: 2024-02-10 06:25:29.514342397 +0000 UTC m=+0.321703883 container died 0ed77770af21057814ee76b70dc9ede38a774e7c854b012d6e878399c56e048d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, architecture=x86_64, vcs-type=git, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, release=754, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-10T06:25:29.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:25:29] ENGINE Bus STOPPING 2024-02-10T06:25:29.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:25:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T06:25:29.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:25:29] ENGINE Bus STOPPED 2024-02-10T06:25:29.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:25:29] ENGINE Bus STARTING 2024-02-10T06:25:29.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:25:29] ENGINE Serving on http://:::9283 2024-02-10T06:25:29.587 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:29 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: [10/Feb/2024:06:25:29] ENGINE Bus STARTED 2024-02-10T06:25:30.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:29 smithi139 podman[87688]: 2024-02-10 06:25:29.858439337 +0000 UTC m=+0.665800869 container cleanup 0ed77770af21057814ee76b70dc9ede38a774e7c854b012d6e878399c56e048d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, ceph=True, GIT_CLEAN=True, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-02-10T06:25:30.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:29 smithi139 bash[87688]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c 2024-02-10T06:25:30.531 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:30 smithi139 podman[87714]: 2024-02-10 06:25:30.162927263 +0000 UTC m=+0.651529695 container remove 0ed77770af21057814ee76b70dc9ede38a774e7c854b012d6e878399c56e048d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, name=centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD) 2024-02-10T06:25:30.531 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:30 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.c.service: Deactivated successfully. 2024-02-10T06:25:30.531 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:30 smithi139 systemd[1]: Stopped Ceph mon.c for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:25:30.531 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:30 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.c.service: Consumed 16.859s CPU time. 2024-02-10T06:25:30.531 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:30 smithi139 systemd[1]: Starting Ceph mon.c for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:25:30.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:30 smithi139 podman[87851]: 2024-02-10 06:25:30.74443139 +0000 UTC m=+0.110806923 container create 2dd65b7a1ca643f2008e2af53d7f004ed545c2b702c95d0379e35947ca92c22e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:25:30.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:30 smithi139 podman[87851]: 2024-02-10 06:25:30.652778251 +0000 UTC m=+0.019153787 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:25:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 podman[87851]: 2024-02-10 06:25:31.013425051 +0000 UTC m=+0.379800584 container init 2dd65b7a1ca643f2008e2af53d7f004ed545c2b702c95d0379e35947ca92c22e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-10T06:25:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 podman[87851]: 2024-02-10 06:25:31.017048202 +0000 UTC m=+0.383423737 container start 2dd65b7a1ca643f2008e2af53d7f004ed545c2b702c95d0379e35947ca92c22e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, CEPH_POINT_RELEASE=) 2024-02-10T06:25:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T06:25:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T06:25:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: pidfile_write: ignore empty --pid-file 2024-02-10T06:25:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: load: jerasure load: lrc 2024-02-10T06:25:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: RocksDB version: 7.9.2 2024-02-10T06:25:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Git sha 0 2024-02-10T06:25:31.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: DB SUMMARY 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: DB Session ID: TDDAXJSCYTNXM06TD2TB 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: CURRENT file: CURRENT 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: MANIFEST file: MANIFEST-000009 size: 3005 Bytes 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000057.sst 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000055.log size: 3148527 ; 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.error_if_exists: 0 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.create_if_missing: 0 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.paranoid_checks: 1 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.env: 0x5615900c1ae0 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T06:25:31.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.info_log: 0x5615909c98c0 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.statistics: (nil) 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.use_fsync: 0 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_log_file_size: 0 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.allow_fallocate: 1 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.use_direct_reads: 0 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T06:25:31.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.db_log_dir: 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.wal_dir: 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.write_buffer_manager: 0x56159099cdc0 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T06:25:31.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T06:25:31.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T06:25:31.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T06:25:31.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T06:25:31.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T06:25:31.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.unordered_write: 0 2024-02-10T06:25:31.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T06:25:31.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T06:25:31.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T06:25:31.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T06:25:31.341 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.row_cache: None 2024-02-10T06:25:31.342 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.wal_filter: None 2024-02-10T06:25:31.342 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T06:25:31.343 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T06:25:31.343 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.two_write_queues: 0 2024-02-10T06:25:31.343 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T06:25:31.343 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.wal_compression: 0 2024-02-10T06:25:31.343 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.atomic_flush: 0 2024-02-10T06:25:31.343 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T06:25:31.343 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T06:25:31.343 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T06:25:31.343 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.log_readahead_size: 0 2024-02-10T06:25:31.343 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T06:25:31.343 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_background_jobs: 2 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_background_compactions: -1 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_subcompactions: 1 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T06:25:31.344 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_open_files: -1 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_background_flushes: -1 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Compression algorithms supported: 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: kZSTD supported: 0 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: kXpressCompression supported: 0 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: kZlibCompression supported: 1 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: kSnappyCompression supported: 1 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: kLZ4Compression supported: 1 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: kBZip2Compression supported: 0 2024-02-10T06:25:31.345 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.merge_operator: 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_filter: None 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5615909c99c0) 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout: cache_index_and_filter_blocks: 1 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout: pin_top_level_index_and_filter: 1 2024-02-10T06:25:31.346 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_type: 0 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: data_block_index_type: 0 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_shortening: 1 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: checksum: 4 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: no_block_cache: 0 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache: 0x5615909cd090 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_name: BinnedLRUCache 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_options: 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: capacity : 536870912 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: num_shard_bits : 4 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: strict_capacity_limit : 0 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: high_pri_pool_ratio: 0.000 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_compressed: (nil) 2024-02-10T06:25:31.347 INFO:journalctl@ceph.mon.c.smithi139.stdout: persistent_cache: (nil) 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_size: 4096 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_size_deviation: 10 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_restart_interval: 16 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_block_restart_interval: 1 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: metadata_block_size: 4096 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: partition_filters: 0 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: use_delta_encoding: 1 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: filter_policy: bloomfilter 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: whole_key_filtering: 1 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: verify_compression: 0 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: read_amp_bytes_per_bit: 0 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: format_version: 5 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: enable_index_compression: 1 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_align: 0 2024-02-10T06:25:31.348 INFO:journalctl@ceph.mon.c.smithi139.stdout: max_auto_readahead_size: 262144 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout: prepopulate_block_cache: 0 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout: initial_auto_readahead_size: 8192 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compression: NoCompression 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.num_levels: 7 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T06:25:31.349 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T06:25:31.350 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T06:25:31.351 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T06:25:31.352 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.table_properties_collectors: 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.inplace_update_support: 0 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.bloom_locality: 0 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.max_successive_merges: 0 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.ttl: 2592000 2024-02-10T06:25:31.353 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.enable_blob_files: false 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.min_blob_size: 0 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 57.sst 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 59, last_sequence is 31726, log_number is 55,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T06:25:31.354 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 55 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 03962755-a1db-4d2d-b0c1-0918fe82ed66 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546331046653, "job": 1, "event": "recovery_started", "wal_files": [55]} 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #55 mode 2 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546331054015, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 60, "file_size": 2022290, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 31732, "largest_seqno": 32438, "table_properties": {"data_size": 2018456, "index_size": 1988, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 837, "raw_key_size": 7624, "raw_average_key_size": 24, "raw_value_size": 2011565, "raw_average_value_size": 6406, "num_data_blocks": 91, "num_entries": 314, "num_filter_entries": 314, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707546331, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "03962755-a1db-4d2d-b0c1-0918fe82ed66", "db_session_id": "TDDAXJSCYTNXM06TD2TB", "orig_file_number": 60, "seqno_to_time_mapping": "N/A"}} 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546331054100, "job": 1, "event": "recovery_finished"} 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/version_set.cc:5047] Creating manifest 62 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000055.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561590abe000 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: DB pointer 0x561590aa8000 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** DB Stats ** 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:25:31.355 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: L0 1/0 1.93 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 577.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: L6 1/0 9.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: Sum 2/0 11.61 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 577.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 577.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-10T06:25:31.356 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** Compaction Stats [default] ** 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 577.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative compaction: 0.00 GB write, 194.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:25:31.357 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval compaction: 0.00 GB write, 194.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout: Block cache BinnedLRUCache@0x5615909cd090#2 capacity: 512.00 MB usage: 2.98 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.8e-05 secs_since: 0 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.91 KB,0.000172853%) IndexBlock(1,2.08 KB,0.000396371%) Misc(1,0.00 KB,0%) 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: starting mon.c rank 1 at public addrs [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] at bind addrs [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: mon.c@-1(???) e3 preinit fsid ce44e742-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: mon.c@-1(???).mds e1 new map 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: mon.c@-1(???).mds e1 print_map 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout: e1 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout: legacy client fscid: -1 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout: No filesystems configured 2024-02-10T06:25:31.358 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: mon.c@-1(???).osd e111 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-10T06:25:31.359 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: mon.c@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:25:31.359 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: mon.c@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:25:31.359 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: mon.c@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:25:31.359 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 ceph-mon[87870]: mon.c@-1(???).paxosservice(auth 1..28) refresh upgraded, format 0 -> 3 2024-02-10T06:25:31.359 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 bash[87851]: 2dd65b7a1ca643f2008e2af53d7f004ed545c2b702c95d0379e35947ca92c22e 2024-02-10T06:25:31.359 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:31 smithi139 systemd[1]: Started Ceph mon.c for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:25:36.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:36 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:25:36.062+0000 7ff625973700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-10T06:25:37.878 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:25:38.557 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:25:38.557 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (2m) 18s ago 24m 19.2M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:25:38.557 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (2m) 18s ago 24m 84.3M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:25:38.557 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (2m) 18s ago 23m 53.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:25:38.557 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (3m) 18s ago 29m 446M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:25:38.557 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (13m) 18s ago 30m 473M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:25:38.557 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (31s) 18s ago 31m 29.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:25:38.557 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (82s) 18s ago 29m 35.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:25:38.557 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 starting - - - 2048M 2024-02-10T06:25:38.557 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (3m) 18s ago 25m 17.3M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:25:38.557 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (3m) 18s ago 25m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:25:38.558 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (28m) 18s ago 28m 54.6M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:25:38.558 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (28m) 18s ago 28m 55.1M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:25:38.558 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (28m) 18s ago 28m 53.5M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:25:38.558 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (27m) 18s ago 27m 53.7M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:25:38.558 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (27m) 18s ago 27m 56.1M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:25:38.558 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (26m) 18s ago 26m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:25:38.558 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (26m) 18s ago 26m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:25:38.558 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (26m) 18s ago 26m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:25:38.558 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (2m) 18s ago 25m 59.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:25:38.558 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (24m) 18s ago 24m 76.6M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:25:38.558 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (23m) 18s ago 23m 75.7M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:25:38.677 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:25:38.677 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:25:38.677 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:25:38.677 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:25:38.677 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: mon.b calling monitor election 2024-02-10T06:25:38.677 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: mon.c calling monitor election 2024-02-10T06:25:38.677 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:38.677 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: fsmap 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: mgrmap e48: y(active, since 22s), standbys: x 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: overall HEALTH_OK 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: mon.a calling monitor election 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: monmap e4: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: fsmap 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: mgrmap e48: y(active, since 22s), standbys: x 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: Cluster is now healthy 2024-02-10T06:25:38.678 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: overall HEALTH_OK 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='' 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='' 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: mon.b calling monitor election 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: mon.c calling monitor election 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:38.679 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: fsmap 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: mgrmap e48: y(active, since 22s), standbys: x 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: overall HEALTH_OK 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: mon.a calling monitor election 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: monmap e4: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: fsmap 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: mgrmap e48: y(active, since 22s), standbys: x 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: Cluster is now healthy 2024-02-10T06:25:38.680 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: overall HEALTH_OK 2024-02-10T06:25:38.681 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='' 2024-02-10T06:25:38.681 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='' 2024-02-10T06:25:38.681 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:38.681 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:38.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T06:25:38.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T06:25:38.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T06:25:38.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T06:25:38.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: mon.b calling monitor election 2024-02-10T06:25:38.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: mon.c calling monitor election 2024-02-10T06:25:38.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:38.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:38.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:38.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:38.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: fsmap 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: mgrmap e48: y(active, since 22s), standbys: x 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: overall HEALTH_OK 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: mon.a calling monitor election 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: monmap e4: 3 mons at {a=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: fsmap 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: mgrmap e48: y(active, since 22s), standbys: x 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: Cluster is now healthy 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: overall HEALTH_OK 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='' 2024-02-10T06:25:38.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='' 2024-02-10T06:25:38.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:38.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:38 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:38.977 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:25:38.977 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:25:38.977 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:25:38.977 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:25:38.977 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:25:38.977 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:25:38.977 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:25:38.977 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:25:38.977 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:25:38.977 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:25:38.977 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:25:38.978 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:25:38.978 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:25:38.978 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:25:38.978 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-10T06:25:38.978 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T06:25:38.978 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:25:38.978 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:25:38.978 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:25:38] "GET /metrics HTTP/1.1" 200 37956 "" "Prometheus/2.43.0" 2024-02-10T06:25:39.313 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:25:39.313 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:25:39.313 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:25:39.313 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi139", 2024-02-10T06:25:39.313 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:25:39.313 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "1/2 daemons upgraded", 2024-02-10T06:25:39.313 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading mon daemons", 2024-02-10T06:25:39.313 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:25:39.314 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:25:39.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[85480]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[85480]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[85480]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/1848138587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:39.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[85480]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[85480]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[87870]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[87870]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[87870]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/1848138587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[87870]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:39 smithi139 ceph-mon[87870]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:39.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:39 smithi150 ceph-mon[68055]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:39 smithi150 ceph-mon[68055]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:39 smithi150 ceph-mon[68055]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:39 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/1848138587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:25:39.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:39 smithi150 ceph-mon[68055]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:25:39.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:39 smithi150 ceph-mon[68055]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:42 smithi139 ceph-mon[85480]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:42 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:42.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:42 smithi139 ceph-mon[87870]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:42.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:42 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:42.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:42 smithi150 ceph-mon[68055]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:42.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:42 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:43.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:43 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:43.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:43 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:43.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:43 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:44.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:44 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:44 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:44 smithi150 ceph-mon[68055]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:44.581 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:44 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:44.581 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:44 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:44.581 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:44 smithi139 ceph-mon[85480]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:44.582 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:44 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:44.582 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:44 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:44.582 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:44 smithi139 ceph-mon[87870]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:46.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[85480]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:46.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:46.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:46.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:46.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:46.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:25:46.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[87870]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:25:46.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:46 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:46.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:46 smithi150 ceph-mon[68055]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:46 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:46 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:46 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:46 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:46 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:25:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:46 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:47.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:47 smithi139 ceph-mon[85480]: Detected new or changed devices on smithi139 2024-02-10T06:25:47.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:47 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:47.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:47 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:47.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:47 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:47.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:47 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:47 smithi139 ceph-mon[87870]: Detected new or changed devices on smithi139 2024-02-10T06:25:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:47 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:47.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:47 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:47 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:47.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:47 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:47.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:47 smithi150 ceph-mon[68055]: Detected new or changed devices on smithi139 2024-02-10T06:25:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:47 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:47 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:47 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:47.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:47 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:48.741 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:25:48] "GET /metrics HTTP/1.1" 200 37949 "" "Prometheus/2.43.0" 2024-02-10T06:25:48.741 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[85480]: Reconfiguring mon.a (monmap changed)... 2024-02-10T06:25:48.741 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[85480]: Reconfiguring daemon mon.a on smithi139 2024-02-10T06:25:48.741 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[85480]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:48.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:48.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:48.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:25:48.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:25:48.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:25:48.742 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:48.742 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[87870]: Reconfiguring mon.a (monmap changed)... 2024-02-10T06:25:48.742 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[87870]: Reconfiguring daemon mon.a on smithi139 2024-02-10T06:25:48.742 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[87870]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:48.742 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:48.743 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:48.743 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:25:48.743 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:25:48.743 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:25:48.743 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:48 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:48.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:48 smithi150 ceph-mon[68055]: Reconfiguring mon.a (monmap changed)... 2024-02-10T06:25:48.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:48 smithi150 ceph-mon[68055]: Reconfiguring daemon mon.a on smithi139 2024-02-10T06:25:48.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:48 smithi150 ceph-mon[68055]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:48.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:48 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:48.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:48 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:48.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:48 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:25:48.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:48 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:25:48.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:48 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:25:48.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:48 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:49.576 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:49 smithi139 ceph-mon[85480]: Reconfiguring mgr.y (monmap changed)... 2024-02-10T06:25:49.576 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:49 smithi139 ceph-mon[85480]: Reconfiguring daemon mgr.y on smithi139 2024-02-10T06:25:49.577 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:49 smithi139 ceph-mon[87870]: Reconfiguring mgr.y (monmap changed)... 2024-02-10T06:25:49.577 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:49 smithi139 ceph-mon[87870]: Reconfiguring daemon mgr.y on smithi139 2024-02-10T06:25:49.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:49 smithi150 ceph-mon[68055]: Reconfiguring mgr.y (monmap changed)... 2024-02-10T06:25:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:49 smithi150 ceph-mon[68055]: Reconfiguring daemon mgr.y on smithi139 2024-02-10T06:25:50.547 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[85480]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:50.547 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:50.547 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:50.547 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:50.547 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:50.547 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:50.548 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[87870]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:50.548 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:50.548 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:50.548 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:50.548 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:50.548 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:50 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:50.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:50 smithi150 ceph-mon[68055]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:50.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:50 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:50.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:50 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:50.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:50 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:25:50.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:50 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:25:50.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:50 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:51.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[85480]: Reconfiguring mon.c (monmap changed)... 2024-02-10T06:25:51.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[85480]: Reconfiguring daemon mon.c on smithi139 2024-02-10T06:25:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:25:51.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:51.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[87870]: Reconfiguring mon.c (monmap changed)... 2024-02-10T06:25:51.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[87870]: Reconfiguring daemon mon.c on smithi139 2024-02-10T06:25:51.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:51.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:51.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:25:51.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:51 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:51.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:51 smithi150 ceph-mon[68055]: Reconfiguring mon.c (monmap changed)... 2024-02-10T06:25:51.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:51 smithi150 ceph-mon[68055]: Reconfiguring daemon mon.c on smithi139 2024-02-10T06:25:51.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:51 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:51.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:51 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:51.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:51 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:25:51.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:51 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:52.798 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:52 smithi139 ceph-mon[87870]: Reconfiguring osd.0 (monmap changed)... 2024-02-10T06:25:52.798 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:52 smithi139 ceph-mon[87870]: Reconfiguring daemon osd.0 on smithi139 2024-02-10T06:25:52.798 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:52 smithi139 ceph-mon[87870]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:52.799 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:52 smithi139 ceph-mon[85480]: Reconfiguring osd.0 (monmap changed)... 2024-02-10T06:25:52.799 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:52 smithi139 ceph-mon[85480]: Reconfiguring daemon osd.0 on smithi139 2024-02-10T06:25:52.799 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:52 smithi139 ceph-mon[85480]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:52.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:52 smithi150 ceph-mon[68055]: Reconfiguring osd.0 (monmap changed)... 2024-02-10T06:25:52.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:52 smithi150 ceph-mon[68055]: Reconfiguring daemon osd.0 on smithi139 2024-02-10T06:25:52.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:52 smithi150 ceph-mon[68055]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[87870]: Reconfiguring osd.1 (monmap changed)... 2024-02-10T06:25:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:25:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[87870]: Reconfiguring daemon osd.1 on smithi139 2024-02-10T06:25:54.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[87870]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:54.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:54.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:54.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[85480]: Reconfiguring osd.1 (monmap changed)... 2024-02-10T06:25:54.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:25:54.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:54.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[85480]: Reconfiguring daemon osd.1 on smithi139 2024-02-10T06:25:54.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:53 smithi139 ceph-mon[85480]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:53 smithi150 ceph-mon[68055]: Reconfiguring osd.1 (monmap changed)... 2024-02-10T06:25:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:25:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:53 smithi150 ceph-mon[68055]: Reconfiguring daemon osd.1 on smithi139 2024-02-10T06:25:54.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:53 smithi150 ceph-mon[68055]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:55.778 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:55.778 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:55.778 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[85480]: Reconfiguring osd.2 (monmap changed)... 2024-02-10T06:25:55.778 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:25:55.779 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:55.779 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[85480]: Reconfiguring daemon osd.2 on smithi139 2024-02-10T06:25:55.779 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[85480]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:55.779 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:55.779 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:55.779 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[87870]: Reconfiguring osd.2 (monmap changed)... 2024-02-10T06:25:55.779 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:25:55.779 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:55.779 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[87870]: Reconfiguring daemon osd.2 on smithi139 2024-02-10T06:25:55.779 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:55 smithi139 ceph-mon[87870]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:55 smithi150 ceph-mon[68055]: Reconfiguring osd.2 (monmap changed)... 2024-02-10T06:25:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:25:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:55 smithi150 ceph-mon[68055]: Reconfiguring daemon osd.2 on smithi139 2024-02-10T06:25:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:55 smithi150 ceph-mon[68055]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:57.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[85480]: Reconfiguring osd.3 (monmap changed)... 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[85480]: Reconfiguring daemon osd.3 on smithi139 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[87870]: Reconfiguring osd.3 (monmap changed)... 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[87870]: Reconfiguring daemon osd.3 on smithi139 2024-02-10T06:25:57.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:57 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:57.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:57 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:57.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:57 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:57.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:57 smithi150 ceph-mon[68055]: Reconfiguring osd.3 (monmap changed)... 2024-02-10T06:25:57.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:57 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:25:57.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:57 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:57.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:57 smithi150 ceph-mon[68055]: Reconfiguring daemon osd.3 on smithi139 2024-02-10T06:25:57.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:57 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:58.386 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[85480]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:58.386 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:58.386 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:58.386 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:25:58.386 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:25:58.386 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:58.386 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[87870]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:58.386 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:58.386 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:58.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:25:58.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:25:58.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:58 smithi150 ceph-mon[68055]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:25:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:25:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:25:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:59.034 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:25:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:25:58] "GET /metrics HTTP/1.1" 200 37949 "" "Prometheus/2.43.0" 2024-02-10T06:25:59.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:59 smithi139 ceph-mon[87870]: Reconfiguring rgw.foo.smithi139.kpyuou (monmap changed)... 2024-02-10T06:25:59.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:25:59 smithi139 ceph-mon[87870]: Reconfiguring daemon rgw.foo.smithi139.kpyuou on smithi139 2024-02-10T06:25:59.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:59 smithi139 ceph-mon[85480]: Reconfiguring rgw.foo.smithi139.kpyuou (monmap changed)... 2024-02-10T06:25:59.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:25:59 smithi139 ceph-mon[85480]: Reconfiguring daemon rgw.foo.smithi139.kpyuou on smithi139 2024-02-10T06:25:59.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:59 smithi150 ceph-mon[68055]: Reconfiguring rgw.foo.smithi139.kpyuou (monmap changed)... 2024-02-10T06:25:59.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:25:59 smithi150 ceph-mon[68055]: Reconfiguring daemon rgw.foo.smithi139.kpyuou on smithi139 2024-02-10T06:26:00.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:00 smithi150 ceph-mon[68055]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:00.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:00 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:00.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:00 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:00.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:00 smithi150 ceph-mon[68055]: Reconfiguring mon.b (monmap changed)... 2024-02-10T06:26:00.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:00 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:26:00.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:00 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:26:00.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:00 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:00.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:00 smithi150 ceph-mon[68055]: Reconfiguring daemon mon.b on smithi150 2024-02-10T06:26:00.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[85480]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:00.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:00.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:00.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[85480]: Reconfiguring mon.b (monmap changed)... 2024-02-10T06:26:00.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:26:00.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:26:00.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:00.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[85480]: Reconfiguring daemon mon.b on smithi150 2024-02-10T06:26:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[87870]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[87870]: Reconfiguring mon.b (monmap changed)... 2024-02-10T06:26:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:26:00.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:26:00.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:00.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:00 smithi139 ceph-mon[87870]: Reconfiguring daemon mon.b on smithi150 2024-02-10T06:26:01.695 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:01.695 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:01.695 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:01.696 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:26:01.696 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:26:01.696 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:26:01.696 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:01.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:01.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:01.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:01.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:26:01.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:26:01.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:26:01.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:26:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:26:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:26:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:02.519 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:02 smithi150 ceph-mon[68055]: Reconfiguring mgr.x (monmap changed)... 2024-02-10T06:26:02.519 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:02 smithi150 ceph-mon[68055]: Reconfiguring daemon mgr.x on smithi150 2024-02-10T06:26:02.519 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:02 smithi150 ceph-mon[68055]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:02.738 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:02 smithi139 ceph-mon[85480]: Reconfiguring mgr.x (monmap changed)... 2024-02-10T06:26:02.738 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:02 smithi139 ceph-mon[85480]: Reconfiguring daemon mgr.x on smithi150 2024-02-10T06:26:02.738 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:02 smithi139 ceph-mon[85480]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:02.738 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:02 smithi139 ceph-mon[87870]: Reconfiguring mgr.x (monmap changed)... 2024-02-10T06:26:02.738 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:02 smithi139 ceph-mon[87870]: Reconfiguring daemon mgr.x on smithi150 2024-02-10T06:26:02.738 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:02 smithi139 ceph-mon[87870]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:03.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:03.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[85480]: Reconfiguring osd.4 (monmap changed)... 2024-02-10T06:26:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:26:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[85480]: Reconfiguring daemon osd.4 on smithi150 2024-02-10T06:26:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[85480]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[87870]: Reconfiguring osd.4 (monmap changed)... 2024-02-10T06:26:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:26:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[87870]: Reconfiguring daemon osd.4 on smithi150 2024-02-10T06:26:03.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:03 smithi139 ceph-mon[87870]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:03.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:03 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:03.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:03 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:03.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:03 smithi150 ceph-mon[68055]: Reconfiguring osd.4 (monmap changed)... 2024-02-10T06:26:03.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:03 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:26:03.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:03 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:03.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:03 smithi150 ceph-mon[68055]: Reconfiguring daemon osd.4 on smithi150 2024-02-10T06:26:03.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:03 smithi150 ceph-mon[68055]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:05.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:05 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:05.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:05 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:05.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:05 smithi150 ceph-mon[68055]: Reconfiguring osd.5 (monmap changed)... 2024-02-10T06:26:05.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:05 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:26:05.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:05 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:05.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:05 smithi150 ceph-mon[68055]: Reconfiguring daemon osd.5 on smithi150 2024-02-10T06:26:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[85480]: Reconfiguring osd.5 (monmap changed)... 2024-02-10T06:26:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:26:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:05.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[85480]: Reconfiguring daemon osd.5 on smithi150 2024-02-10T06:26:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:05.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[87870]: Reconfiguring osd.5 (monmap changed)... 2024-02-10T06:26:05.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:26:05.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:05.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:05 smithi139 ceph-mon[87870]: Reconfiguring daemon osd.5 on smithi150 2024-02-10T06:26:06.434 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:06 smithi150 ceph-mon[68055]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:06.435 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:06 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:06.435 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:06 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:06.435 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:06 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:26:06.435 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:06 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:06.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:06 smithi139 ceph-mon[87870]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:06 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:06 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:06 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:26:06.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:06 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:06 smithi139 ceph-mon[85480]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:06 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:06 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:06 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:26:06.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:06 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:07 smithi139 ceph-mon[85480]: Reconfiguring osd.6 (monmap changed)... 2024-02-10T06:26:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:07 smithi139 ceph-mon[85480]: Reconfiguring daemon osd.6 on smithi150 2024-02-10T06:26:07.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:07 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:07 smithi139 ceph-mon[87870]: Reconfiguring osd.6 (monmap changed)... 2024-02-10T06:26:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:07 smithi139 ceph-mon[87870]: Reconfiguring daemon osd.6 on smithi150 2024-02-10T06:26:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:07 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:07 smithi150 ceph-mon[68055]: Reconfiguring osd.6 (monmap changed)... 2024-02-10T06:26:07.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:07 smithi150 ceph-mon[68055]: Reconfiguring daemon osd.6 on smithi150 2024-02-10T06:26:07.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:07 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:08 smithi150 ceph-mon[68055]: Reconfiguring osd.7 (monmap changed)... 2024-02-10T06:26:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:26:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:08 smithi150 ceph-mon[68055]: Reconfiguring daemon osd.7 on smithi150 2024-02-10T06:26:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:08 smithi150 ceph-mon[68055]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:08.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[87870]: Reconfiguring osd.7 (monmap changed)... 2024-02-10T06:26:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:26:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[87870]: Reconfiguring daemon osd.7 on smithi150 2024-02-10T06:26:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[87870]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[85480]: Reconfiguring osd.7 (monmap changed)... 2024-02-10T06:26:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:26:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:08.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[85480]: Reconfiguring daemon osd.7 on smithi150 2024-02-10T06:26:08.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-mon[85480]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:09.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:26:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:26:08] "GET /metrics HTTP/1.1" 200 37953 "" "Prometheus/2.43.0" 2024-02-10T06:26:09.672 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:26:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[87870]: Reconfiguring rgw.foo.smithi150.subshb (monmap changed)... 2024-02-10T06:26:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:26:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:26:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[87870]: Reconfiguring daemon rgw.foo.smithi150.subshb on smithi150 2024-02-10T06:26:10.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[87870]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:10.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:10.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:10.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[85480]: Reconfiguring rgw.foo.smithi150.subshb (monmap changed)... 2024-02-10T06:26:10.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:26:10.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:26:10.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:10.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[85480]: Reconfiguring daemon rgw.foo.smithi150.subshb on smithi150 2024-02-10T06:26:10.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:09 smithi139 ceph-mon[85480]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:09 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:09 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:10.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:09 smithi150 ceph-mon[68055]: Reconfiguring rgw.foo.smithi150.subshb (monmap changed)... 2024-02-10T06:26:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:09 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:26:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:09 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:26:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:09 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:09 smithi150 ceph-mon[68055]: Reconfiguring daemon rgw.foo.smithi150.subshb on smithi150 2024-02-10T06:26:10.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:09 smithi150 ceph-mon[68055]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:10.320 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:26:10.320 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (3m) 28s ago 25m 19.2M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (2m) 50s ago 25m 84.3M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (3m) 28s ago 24m 54.6M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (4m) 50s ago 29m 446M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (14m) 28s ago 31m 492M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (63s) 28s ago 31m 42.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (113s) 50s ago 29m 35.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (39s) 28s ago 30m 22.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (3m) 28s ago 25m 17.0M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (3m) 50s ago 25m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (29m) 28s ago 29m 54.7M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (29m) 28s ago 28m 55.4M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (28m) 28s ago 28m 53.4M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (28m) 28s ago 28m 53.7M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:26:10.321 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (27m) 50s ago 27m 56.1M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:26:10.322 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (27m) 50s ago 27m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:26:10.322 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (27m) 50s ago 27m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:26:10.322 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (26m) 50s ago 26m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:26:10.322 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (3m) 50s ago 26m 59.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:26:10.322 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (24m) 28s ago 24m 76.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:26:10.322 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (24m) 50s ago 24m 75.7M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:26:10.742 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:26:10.742 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:26:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-10T06:26:10.744 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T06:26:10.744 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:26:10.744 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:26:11.074 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:26:11.074 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": null, 2024-02-10T06:26:11.074 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": false, 2024-02-10T06:26:11.074 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "", 2024-02-10T06:26:11.074 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:26:11.074 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": null, 2024-02-10T06:26:11.074 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "", 2024-02-10T06:26:11.074 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:26:11.075 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:26:11.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:11.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:11.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='client.64114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:11.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:11.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-10T06:26:11.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-10T06:26:11.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.841 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.842 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.843 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.844 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.844 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.844 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.844 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.844 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='client.64114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:26:11.847 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:26:11.848 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:26:11.848 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:26:11.848 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:26:11.848 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:26:11.848 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:26:11.848 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:26:11.848 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:26:11.848 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:26:11.848 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.849 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:26:11.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:26:11.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:26:11.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:26:11.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:26:11.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:26:11.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:26:11.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:26:11.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.852 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/1106816761' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:11.853 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:26:11.854 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:26:11.855 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:26:11.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/1106816761' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:26:11.858 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:11.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:11.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='client.64114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:11.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-10T06:26:11.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:26:11.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:26:11.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:26:11.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:26:11.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:26:11.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:26:11.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:26:11.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:26:11.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:26:11.882 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.883 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/1106816761' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:26:11.884 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:12.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mon 2024-02-10T06:26:12.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all crash 2024-02-10T06:26:12.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mds 2024-02-10T06:26:12.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:26:12.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:26:12.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:26:12.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all nfs 2024-02-10T06:26:12.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:26:12.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all node-exporter 2024-02-10T06:26:12.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all prometheus 2024-02-10T06:26:12.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all alertmanager 2024-02-10T06:26:12.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all grafana 2024-02-10T06:26:12.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all loki 2024-02-10T06:26:12.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all promtail 2024-02-10T06:26:12.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Finalizing container_image settings 2024-02-10T06:26:12.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: Upgrade: Complete! 2024-02-10T06:26:12.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:12.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[85480]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:12.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mon 2024-02-10T06:26:12.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all crash 2024-02-10T06:26:12.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mds 2024-02-10T06:26:12.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:26:12.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:26:12.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:26:12.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all nfs 2024-02-10T06:26:12.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:26:12.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all node-exporter 2024-02-10T06:26:12.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all prometheus 2024-02-10T06:26:12.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all alertmanager 2024-02-10T06:26:12.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all grafana 2024-02-10T06:26:12.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all loki 2024-02-10T06:26:12.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all promtail 2024-02-10T06:26:12.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Finalizing container_image settings 2024-02-10T06:26:12.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: Upgrade: Complete! 2024-02-10T06:26:12.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:12.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:12 smithi139 ceph-mon[87870]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mon 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all crash 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mds 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all nfs 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all node-exporter 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all prometheus 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all alertmanager 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all grafana 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all loki 2024-02-10T06:26:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all promtail 2024-02-10T06:26:12.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Finalizing container_image settings 2024-02-10T06:26:12.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: Upgrade: Complete! 2024-02-10T06:26:12.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:12.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:12 smithi150 ceph-mon[68055]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:14.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:14 smithi139 ceph-mon[85480]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:14.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:14 smithi139 ceph-mon[87870]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:14.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:14 smithi150 ceph-mon[68055]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:16.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:16 smithi139 ceph-mon[85480]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:16.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:16.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:16.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:16 smithi139 ceph-mon[87870]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:16.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:16 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:16.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:16 smithi150 ceph-mon[68055]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:26:17.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:17 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:17.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:17 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:17.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:17 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:18.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:18 smithi139 ceph-mon[85480]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:18.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:18 smithi139 ceph-mon[87870]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:18.837 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:26:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:26:18] "GET /metrics HTTP/1.1" 200 37885 "" "Prometheus/2.43.0" 2024-02-10T06:26:18.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:18 smithi150 ceph-mon[68055]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:20.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:20 smithi139 ceph-mon[85480]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:20 smithi139 ceph-mon[87870]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:20.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:20 smithi150 ceph-mon[68055]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:22.822 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:22 smithi150 ceph-mon[68055]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:22.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:22 smithi139 ceph-mon[85480]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:22.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:22 smithi139 ceph-mon[87870]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:24.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:24 smithi139 ceph-mon[85480]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:24.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:24 smithi139 ceph-mon[87870]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:24.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:24 smithi150 ceph-mon[68055]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:26.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:26 smithi139 ceph-mon[85480]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:26.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:26 smithi139 ceph-mon[87870]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:26.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:26 smithi150 ceph-mon[68055]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:27.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:27 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:27 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:27.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:27 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:28.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:28 smithi139 ceph-mon[85480]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:28.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:28 smithi139 ceph-mon[87870]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:28.837 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:26:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:26:28] "GET /metrics HTTP/1.1" 200 37885 "" "Prometheus/2.43.0" 2024-02-10T06:26:28.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:28 smithi150 ceph-mon[68055]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:30.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:30 smithi139 ceph-mon[85480]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:30 smithi139 ceph-mon[87870]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:30.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:30 smithi150 ceph-mon[68055]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:32.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:32 smithi150 ceph-mon[68055]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:32.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:32 smithi139 ceph-mon[85480]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:32.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:32 smithi139 ceph-mon[87870]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:34.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:34 smithi139 ceph-mon[85480]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:34.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:34 smithi139 ceph-mon[87870]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:34.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:34 smithi150 ceph-mon[68055]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:36.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:36 smithi139 ceph-mon[85480]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:36.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:36 smithi139 ceph-mon[87870]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:36.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:36 smithi150 ceph-mon[68055]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:37.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:37 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:37.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:37 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:37.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:37 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:38.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:38 smithi139 ceph-mon[85480]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:38.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:38 smithi139 ceph-mon[87870]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:38.837 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:26:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:26:38] "GET /metrics HTTP/1.1" 200 37950 "" "Prometheus/2.43.0" 2024-02-10T06:26:38.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:38 smithi150 ceph-mon[68055]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:40.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:40 smithi139 ceph-mon[85480]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:40.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:40 smithi139 ceph-mon[87870]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:40.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:40 smithi150 ceph-mon[68055]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:42.150 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:26:42.712 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:42 smithi139 ceph-mon[85480]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:42.712 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:42 smithi139 ceph-mon[85480]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:42.712 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:42 smithi139 ceph-mon[87870]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:42.712 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:42 smithi139 ceph-mon[87870]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:42.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:42 smithi150 ceph-mon[68055]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:42 smithi150 ceph-mon[68055]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:44.176 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:26:44.176 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (3m) 62s ago 25m 19.2M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:26:44.176 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (3m) 84s ago 25m 84.3M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:26:44.176 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (3m) 62s ago 24m 54.6M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:26:44.176 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (4m) 84s ago 30m 446M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:26:44.176 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (14m) 62s ago 31m 492M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:26:44.176 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (97s) 62s ago 32m 42.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:26:44.176 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (2m) 84s ago 30m 35.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:26:44.176 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (73s) 62s ago 30m 22.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:26:44.176 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (4m) 62s ago 26m 17.0M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:26:44.176 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (4m) 84s ago 26m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:26:44.177 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (29m) 62s ago 29m 54.7M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:26:44.177 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (29m) 62s ago 29m 55.4M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:26:44.177 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (29m) 62s ago 29m 53.4M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:26:44.177 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (28m) 62s ago 28m 53.7M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:26:44.177 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (28m) 84s ago 28m 56.1M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:26:44.177 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (28m) 84s ago 28m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:26:44.177 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (27m) 84s ago 27m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:26:44.177 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (27m) 84s ago 27m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:26:44.177 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (4m) 84s ago 26m 59.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:26:44.177 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (25m) 62s ago 25m 76.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:26:44.177 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (25m) 84s ago 25m 75.7M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:26:44.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:44 smithi139 ceph-mon[85480]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:44.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:44 smithi139 ceph-mon[87870]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:44 smithi150 ceph-mon[68055]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:44.898 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 1'"'"'' 2024-02-10T06:26:45.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:45 smithi139 ceph-mon[85480]: from='client.64138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:45.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:45 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:45.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:45 smithi139 ceph-mon[87870]: from='client.64138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:45.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:45 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:45.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:45 smithi150 ceph-mon[68055]: from='client.64138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:45.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:45 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:46.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:46 smithi139 ceph-mon[87870]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:46 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:46.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:46 smithi139 ceph-mon[85480]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:46.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:46 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:46.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:46 smithi150 ceph-mon[68055]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:46 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:47.057 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:26:47.796 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.mon | keys'"'"' | grep $sha1' 2024-02-10T06:26:48.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:47 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/1440759717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:48.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:47 smithi139 ceph-mon[85480]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:48.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:47 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/1440759717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:48.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:47 smithi139 ceph-mon[87870]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:48.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:47 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/1440759717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:47 smithi150 ceph-mon[68055]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:49.019 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:26:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:26:48] "GET /metrics HTTP/1.1" 200 37953 "" "Prometheus/2.43.0" 2024-02-10T06:26:49.798 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)" 2024-02-10T06:26:50.519 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:50 smithi139 ceph-mon[85480]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:50.519 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:50 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/691595540' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:50.519 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:50 smithi139 ceph-mon[87870]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:50.519 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:50 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/691595540' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:50.520 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 5'"'"'' 2024-02-10T06:26:50.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:50 smithi150 ceph-mon[68055]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:50.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:50 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/691595540' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:26:52.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:52 smithi139 ceph-mon[85480]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:52.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:52 smithi139 ceph-mon[87870]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:52.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:52 smithi150 ceph-mon[68055]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:54.633 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:26:54.756 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:54 smithi139 ceph-mon[85480]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:54.757 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:54 smithi139 ceph-mon[85480]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:54.757 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:54 smithi139 ceph-mon[87870]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:54.757 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:54 smithi139 ceph-mon[87870]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:54.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:54 smithi150 ceph-mon[68055]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:54.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:54 smithi150 ceph-mon[68055]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:55.247 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade status' 2024-02-10T06:26:56.698 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:56 smithi139 ceph-mon[85480]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:56.698 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:56 smithi139 ceph-mon[87870]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:56.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:56 smithi150 ceph-mon[68055]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:57.190 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:26:57.190 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": null, 2024-02-10T06:26:57.190 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": false, 2024-02-10T06:26:57.190 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "", 2024-02-10T06:26:57.190 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:26:57.190 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": null, 2024-02-10T06:26:57.191 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "", 2024-02-10T06:26:57.191 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:26:57.191 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:26:57.762 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:57 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:57.762 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:57 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:57.786 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T06:26:57.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:57 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:58.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:58 smithi139 ceph-mon[85480]: from='client.64162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:58.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:26:58 smithi139 ceph-mon[85480]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:58.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:58 smithi139 ceph-mon[87870]: from='client.64162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:58.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:26:58 smithi139 ceph-mon[87870]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:58.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:58 smithi150 ceph-mon[68055]: from='client.64162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:26:58.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:26:58 smithi150 ceph-mon[68055]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:59.014 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:26:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:26:58] "GET /metrics HTTP/1.1" 200 37953 "" "Prometheus/2.43.0" 2024-02-10T06:26:59.833 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_OK 2024-02-10T06:27:00.412 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2' 2024-02-10T06:27:00.721 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:00 smithi139 ceph-mon[85480]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:00.721 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:00 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/481903824' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:27:00.721 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:00 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:00.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:00 smithi139 ceph-mon[87870]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:00.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:00 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/481903824' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:27:00.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:00 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:00.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:00 smithi150 ceph-mon[68055]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:00.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:00 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/481903824' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:27:00.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:00 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:02.592 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:02 smithi139 ceph-mon[85480]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:02.592 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:02 smithi139 ceph-mon[87870]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:02.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:02 smithi150 ceph-mon[68055]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:03.660 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:03 smithi139 ceph-mon[85480]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:03.660 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:03 smithi139 ceph-mon[87870]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:03.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:03 smithi150 ceph-mon[68055]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:04.581 INFO:teuthology.orchestra.run.smithi139.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:27:04.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:04 smithi139 ceph-mon[85480]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:04.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:04 smithi139 ceph-mon[87870]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:04.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:04 smithi150 ceph-mon[68055]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:05.194 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T06:27:05.797 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[87870]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:27:05.797 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.797 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:05.797 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[87870]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[85480]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.798 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:05 smithi139 ceph-mon[85480]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:05.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:05 smithi150 ceph-mon[68055]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:27:05.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:05 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:05 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:05.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:05 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:05 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:05 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:05 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:05.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:05 smithi150 ceph-mon[68055]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:06.721 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:06 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:06.721 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:06 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:27:06.721 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:06 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:06.721 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:06 smithi139 ceph-mon[85480]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:27:06.721 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:06 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:06.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:06 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:06.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:06 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:27:06.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:06 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:06.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:06 smithi139 ceph-mon[87870]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:27:06.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:06 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:06 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:06 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:27:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:06 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:06 smithi150 ceph-mon[68055]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:27:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:06 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:07.234 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:27:07.900 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:27:07.900 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (4m) 85s ago 26m 19.2M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:27:07.900 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (3m) 108s ago 25m 84.3M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:27:07.900 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (4m) 85s ago 25m 54.6M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:27:07.900 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (5m) 108s ago 30m 446M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:27:07.900 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (15m) 85s ago 32m 492M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (2m) 85s ago 32m 42.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (2m) 108s ago 30m 35.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (97s) 85s ago 31m 22.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (4m) 85s ago 26m 17.0M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (4m) 108s ago 26m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (30m) 85s ago 30m 54.7M 2992M 17.2.0 e1d6a67b021e 10f8e85c42ee 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (29m) 85s ago 29m 55.4M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (29m) 85s ago 29m 53.4M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (29m) 85s ago 29m 53.7M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (28m) 108s ago 28m 56.1M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (28m) 108s ago 28m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (28m) 108s ago 28m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (27m) 108s ago 27m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (4m) 108s ago 27m 59.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (25m) 85s ago 25m 76.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:27:07.901 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (25m) 108s ago 25m 75.7M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:27:08.281 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:08 smithi139 ceph-mon[85480]: from='client.54246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:08.281 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:08 smithi139 ceph-mon[85480]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:08.281 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:08 smithi139 ceph-mon[87870]: from='client.54246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:08.281 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:08 smithi139 ceph-mon[87870]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:08.295 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:27:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:27:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:27:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:27:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:27:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:27:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:27:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:27:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:27:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:27:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:27:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:27:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:27:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:27:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-10T06:27:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T06:27:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:27:08.296 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:27:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:08 smithi150 ceph-mon[68055]: from='client.54246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:08 smithi150 ceph-mon[68055]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:08.646 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:27:08.646 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:27:08.646 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:27:08.647 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (2 remaining).", 2024-02-10T06:27:08.647 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:27:08.647 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "0/8 daemons upgraded", 2024-02-10T06:27:08.647 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:27:08.647 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:27:08.647 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:27:08.939 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:27:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:27:08] "GET /metrics HTTP/1.1" 200 37948 "" "Prometheus/2.43.0" 2024-02-10T06:27:09.234 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:09.234 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='client.54258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:09.234 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/628049901' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.234 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.234 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:27:09.234 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:27:09.234 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mgr 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mon 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all crash 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T06:27:09.235 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[85480]: Upgrade: osd.0 is safe to restart 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='client.54258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/628049901' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mgr 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mon 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.236 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all crash 2024-02-10T06:27:09.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T06:27:09.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T06:27:09.237 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:09 smithi139 ceph-mon[87870]: Upgrade: osd.0 is safe to restart 2024-02-10T06:27:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='client.54258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/628049901' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:27:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:27:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mgr 2024-02-10T06:27:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mon 2024-02-10T06:27:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all crash 2024-02-10T06:27:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T06:27:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T06:27:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:09 smithi150 ceph-mon[68055]: Upgrade: osd.0 is safe to restart 2024-02-10T06:27:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:10 smithi139 ceph-mon[85480]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:10 smithi139 ceph-mon[85480]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:10 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:10.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:10 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:27:10.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:10 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:10 smithi139 ceph-mon[87870]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:10 smithi139 ceph-mon[87870]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:10 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:10 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:27:10.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:10 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:10 smithi150 ceph-mon[68055]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:10 smithi150 ceph-mon[68055]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:10 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:10 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:27:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:10 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:11.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:11 smithi139 ceph-mon[85480]: Upgrade: Updating osd.0 2024-02-10T06:27:11.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:11 smithi139 ceph-mon[85480]: Deploying daemon osd.0 on smithi139 2024-02-10T06:27:11.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:11 smithi139 ceph-mon[87870]: Upgrade: Updating osd.0 2024-02-10T06:27:11.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:11 smithi139 ceph-mon[87870]: Deploying daemon osd.0 on smithi139 2024-02-10T06:27:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:11 smithi150 ceph-mon[68055]: Upgrade: Updating osd.0 2024-02-10T06:27:11.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:11 smithi150 ceph-mon[68055]: Deploying daemon osd.0 on smithi139 2024-02-10T06:27:12.216 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:12 smithi139 ceph-mon[85480]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:12.217 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:12 smithi139 ceph-mon[87870]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:12.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:12 smithi150 ceph-mon[68055]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:13.837 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:13 smithi139 systemd[1]: Stopping Ceph osd.0 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:27:13.837 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[32312]: 2024-02-10T06:27:13.531+0000 7f6cf4d56700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:27:13.837 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[32312]: 2024-02-10T06:27:13.531+0000 7f6cf4d56700 -1 osd.0 111 *** Got signal Terminated *** 2024-02-10T06:27:13.837 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[32312]: 2024-02-10T06:27:13.531+0000 7f6cf4d56700 -1 osd.0 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:27:14.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:14 smithi139 ceph-mon[85480]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:14.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:14 smithi139 ceph-mon[85480]: osd.0 marked itself down and dead 2024-02-10T06:27:14.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:14 smithi139 ceph-mon[87870]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:14.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:14 smithi139 ceph-mon[87870]: osd.0 marked itself down and dead 2024-02-10T06:27:14.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:14 smithi150 ceph-mon[68055]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:14.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:14 smithi150 ceph-mon[68055]: osd.0 marked itself down and dead 2024-02-10T06:27:15.200 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:14 smithi139 podman[95073]: 2024-02-10 06:27:14.911407793 +0000 UTC m=+1.498230355 container died 10f8e85c42ee06baf6de4e147805e2bf9623e977d8491cf938f366e21d598ad6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, release=754, architecture=x86_64, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., version=8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-type=git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD) 2024-02-10T06:27:15.466 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:15 smithi139 podman[95073]: 2024-02-10 06:27:15.273322805 +0000 UTC m=+1.860145367 container cleanup 10f8e85c42ee06baf6de4e147805e2bf9623e977d8491cf938f366e21d598ad6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0, version=8, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, architecture=x86_64, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux ) 2024-02-10T06:27:15.466 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:15 smithi139 bash[95073]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0 2024-02-10T06:27:15.836 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:15 smithi139 podman[95091]: 2024-02-10 06:27:15.574812268 +0000 UTC m=+0.665567034 container remove 10f8e85c42ee06baf6de4e147805e2bf9623e977d8491cf938f366e21d598ad6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.openshift.expose-services=, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: osdmap e112: 8 total, 7 up, 8 in 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 4]}]: dispatch 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 4]}]: dispatch 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [3, 6]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 0]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [3, 6]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.11", "id": [3, 1]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 0]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.11", "id": [3, 1]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 0]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0, 5, 1]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 0]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0, 5, 1]}]: dispatch 2024-02-10T06:27:15.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: osdmap e112: 8 total, 7 up, 8 in 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 4]}]: dispatch 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 4]}]: dispatch 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [3, 6]}]: dispatch 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 0]}]: dispatch 2024-02-10T06:27:15.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [3, 6]}]: dispatch 2024-02-10T06:27:15.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.11", "id": [3, 1]}]: dispatch 2024-02-10T06:27:15.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 0]}]: dispatch 2024-02-10T06:27:15.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.11", "id": [3, 1]}]: dispatch 2024-02-10T06:27:15.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 0]}]: dispatch 2024-02-10T06:27:15.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0, 5, 1]}]: dispatch 2024-02-10T06:27:15.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 0]}]: dispatch 2024-02-10T06:27:15.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:15 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0, 5, 1]}]: dispatch 2024-02-10T06:27:15.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:27:15.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: osdmap e112: 8 total, 7 up, 8 in 2024-02-10T06:27:15.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: failed to encode map e112 with expected crc 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 4]}]: dispatch 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 4]}]: dispatch 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [3, 6]}]: dispatch 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 0]}]: dispatch 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [3, 6]}]: dispatch 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.11", "id": [3, 1]}]: dispatch 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 0]}]: dispatch 2024-02-10T06:27:15.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.11", "id": [3, 1]}]: dispatch 2024-02-10T06:27:15.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 0]}]: dispatch 2024-02-10T06:27:15.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]: dispatch 2024-02-10T06:27:15.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0, 5, 1]}]: dispatch 2024-02-10T06:27:15.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 0]}]: dispatch 2024-02-10T06:27:15.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:15 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0, 5, 1]}]: dispatch 2024-02-10T06:27:16.162 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:15 smithi139 podman[95178]: 2024-02-10 06:27:15.790251178 +0000 UTC m=+0.018387929 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:27:16.162 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:15 smithi139 podman[95178]: 2024-02-10 06:27:15.901153218 +0000 UTC m=+0.129290002 container create 14c645a721bd927b9effdc38396b21340261b84c52720e1a9598a6b23da6e935 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:27:16.162 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:16 smithi139 podman[95178]: 2024-02-10 06:27:16.16232389 +0000 UTC m=+0.390460642 container init 14c645a721bd927b9effdc38396b21340261b84c52720e1a9598a6b23da6e935 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-10T06:27:16.460 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:16 smithi139 podman[95178]: 2024-02-10 06:27:16.165735506 +0000 UTC m=+0.393872254 container start 14c645a721bd927b9effdc38396b21340261b84c52720e1a9598a6b23da6e935 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-10T06:27:16.461 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:16 smithi139 podman[95178]: 2024-02-10 06:27:16.237113741 +0000 UTC m=+0.465250495 container attach 14c645a721bd927b9effdc38396b21340261b84c52720e1a9598a6b23da6e935 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-10T06:27:16.749 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:16 smithi139 podman[95178]: 2024-02-10 06:27:16.460931822 +0000 UTC m=+0.689068578 container died 14c645a721bd927b9effdc38396b21340261b84c52720e1a9598a6b23da6e935 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-10T06:27:16.749 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: pgmap v64: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:27:16.749 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 4]}]': finished 2024-02-10T06:27:16.749 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-02-10T06:27:16.749 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]': finished 2024-02-10T06:27:16.749 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [3, 6]}]': finished 2024-02-10T06:27:16.749 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 0]}]': finished 2024-02-10T06:27:16.749 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.11", "id": [3, 1]}]': finished 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]': finished 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 0]}]': finished 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0, 5, 1]}]': finished 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: osdmap e113: 8 total, 7 up, 8 in 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.750 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: pgmap v64: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 4]}]': finished 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]': finished 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [3, 6]}]': finished 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 0]}]': finished 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.11", "id": [3, 1]}]': finished 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]': finished 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 0]}]': finished 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0, 5, 1]}]': finished 2024-02-10T06:27:16.751 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.752 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.752 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: osdmap e113: 8 total, 7 up, 8 in 2024-02-10T06:27:16.752 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.752 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.752 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.752 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.752 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.752 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.752 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.752 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:16.752 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:16 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: pgmap v64: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 4]}]': finished 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]': finished 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [3, 6]}]': finished 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 0]}]': finished 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.11", "id": [3, 1]}]': finished 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]': finished 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 0]}]': finished 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0, 5, 1]}]': finished 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: osdmap e113: 8 total, 7 up, 8 in 2024-02-10T06:27:16.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: failed to encode map e113 with expected crc 2024-02-10T06:27:16.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:16.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:17.291 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:17 smithi139 podman[95204]: 2024-02-10 06:27:17.040997043 +0000 UTC m=+0.574972792 container remove 14c645a721bd927b9effdc38396b21340261b84c52720e1a9598a6b23da6e935 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:27:17.291 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:17 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-10T06:27:17.291 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:17 smithi139 systemd[1]: Stopped Ceph osd.0 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:27:17.291 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:17 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.0.service: Consumed 14.059s CPU time. 2024-02-10T06:27:17.291 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:17 smithi139 systemd[1]: Starting Ceph osd.0 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:27:17.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[85480]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[85480]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:17.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[85480]: osdmap e114: 8 total, 7 up, 8 in 2024-02-10T06:27:17.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[85480]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[85480]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[85480]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[85480]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[87870]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[87870]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:17.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[87870]: osdmap e114: 8 total, 7 up, 8 in 2024-02-10T06:27:17.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[87870]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[87870]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[87870]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:17 smithi139 ceph-mon[87870]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.588 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:17 smithi139 podman[95306]: 2024-02-10 06:27:17.423605663 +0000 UTC m=+0.019279364 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:27:17.588 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:17 smithi139 podman[95306]: 2024-02-10 06:27:17.542640964 +0000 UTC m=+0.138314661 container create 3b21f6fc76f625aa2d208385cc675b40da961b5a087893fc438befe1f9f09d5d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:27:17.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:17 smithi150 ceph-mon[68055]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:17 smithi150 ceph-mon[68055]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:17 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:17.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:17 smithi150 ceph-mon[68055]: osdmap e114: 8 total, 7 up, 8 in 2024-02-10T06:27:17.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:17 smithi150 ceph-mon[68055]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:17 smithi150 ceph-mon[68055]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:17 smithi150 ceph-mon[68055]: failed to encode map e114 with expected crc 2024-02-10T06:27:17.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:17 smithi150 ceph-mon[68055]: failed to encode map e114 with expected crc 2024-02-10T06:27:18.086 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:17 smithi139 podman[95306]: 2024-02-10 06:27:17.828643096 +0000 UTC m=+0.424316793 container init 3b21f6fc76f625aa2d208385cc675b40da961b5a087893fc438befe1f9f09d5d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS) 2024-02-10T06:27:18.087 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:17 smithi139 podman[95306]: 2024-02-10 06:27:17.831944218 +0000 UTC m=+0.427617915 container start 3b21f6fc76f625aa2d208385cc675b40da961b5a087893fc438befe1f9f09d5d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2) 2024-02-10T06:27:18.087 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:17 smithi139 podman[95306]: 2024-02-10 06:27:17.903675314 +0000 UTC m=+0.499349010 container attach 3b21f6fc76f625aa2d208385cc675b40da961b5a087893fc438befe1f9f09d5d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:27:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[85480]: pgmap v67: 161 pgs: 6 peering, 21 active+undersized+degraded, 34 active+undersized, 100 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%) 2024-02-10T06:27:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[85480]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[85480]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[85480]: osdmap e115: 8 total, 7 up, 8 in 2024-02-10T06:27:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[85480]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[85480]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[85480]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[85480]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[85480]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-10T06:27:18.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[85480]: Health check failed: Degraded data redundancy: 67/627 objects degraded (10.686%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T06:27:18.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[87870]: pgmap v67: 161 pgs: 6 peering, 21 active+undersized+degraded, 34 active+undersized, 100 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%) 2024-02-10T06:27:18.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[87870]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[87870]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[87870]: osdmap e115: 8 total, 7 up, 8 in 2024-02-10T06:27:18.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[87870]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[87870]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[87870]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[87870]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[87870]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-10T06:27:18.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-mon[87870]: Health check failed: Degraded data redundancy: 67/627 objects degraded (10.686%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate[95329]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate[95329]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 bash[95306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 bash[95306]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 bash[95306]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 bash[95306]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 bash[95306]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 bash[95306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 bash[95306]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate[95329]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate[95329]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate[95329]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate[95329]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T06:27:18.589 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate[95329]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-10T06:27:18.590 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 podman[95306]: 2024-02-10 06:27:18.305787863 +0000 UTC m=+0.901461560 container died 3b21f6fc76f625aa2d208385cc675b40da961b5a087893fc438befe1f9f09d5d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-10T06:27:18.873 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:27:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:27:18] "GET /metrics HTTP/1.1" 200 38012 "" "Prometheus/2.43.0" 2024-02-10T06:27:18.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:18 smithi150 ceph-mon[68055]: pgmap v67: 161 pgs: 6 peering, 21 active+undersized+degraded, 34 active+undersized, 100 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%) 2024-02-10T06:27:18.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:18 smithi150 ceph-mon[68055]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:18 smithi150 ceph-mon[68055]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:18 smithi150 ceph-mon[68055]: osdmap e115: 8 total, 7 up, 8 in 2024-02-10T06:27:18.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:18 smithi150 ceph-mon[68055]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:18 smithi150 ceph-mon[68055]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:18 smithi150 ceph-mon[68055]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:18 smithi150 ceph-mon[68055]: failed to encode map e115 with expected crc 2024-02-10T06:27:18.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:18 smithi150 ceph-mon[68055]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-10T06:27:18.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:18 smithi150 ceph-mon[68055]: Health check failed: Degraded data redundancy: 67/627 objects degraded (10.686%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T06:27:19.219 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:18 smithi139 podman[95452]: 2024-02-10 06:27:18.873791771 +0000 UTC m=+0.563041012 container remove 3b21f6fc76f625aa2d208385cc675b40da961b5a087893fc438befe1f9f09d5d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=) 2024-02-10T06:27:19.219 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:19 smithi139 podman[95514]: 2024-02-10 06:27:18.98999966 +0000 UTC m=+0.017774927 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:27:19.219 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:19 smithi139 podman[95514]: 2024-02-10 06:27:19.091473754 +0000 UTC m=+0.119249058 container create 8287393c3c4f1b22711aec71463b2a9a45ffcbf65302025284a83ae1cdce53de (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True) 2024-02-10T06:27:19.586 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:19 smithi139 podman[95514]: 2024-02-10 06:27:19.328058377 +0000 UTC m=+0.355833643 container init 8287393c3c4f1b22711aec71463b2a9a45ffcbf65302025284a83ae1cdce53de (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True) 2024-02-10T06:27:19.587 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:19 smithi139 podman[95514]: 2024-02-10 06:27:19.331025813 +0000 UTC m=+0.358801081 container start 8287393c3c4f1b22711aec71463b2a9a45ffcbf65302025284a83ae1cdce53de (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-02-10T06:27:19.587 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:19 smithi139 ceph-osd[95533]: -- 172.21.15.139:0/2929409304 <== mon.1 v2:172.21.15.139:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560a6b4691e0 con 0x560a6b43b000 2024-02-10T06:27:19.587 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:19 smithi139 bash[95514]: 8287393c3c4f1b22711aec71463b2a9a45ffcbf65302025284a83ae1cdce53de 2024-02-10T06:27:19.587 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:19 smithi139 systemd[1]: Started Ceph osd.0 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:27:20.283 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:19 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[95528]: 2024-02-10T06:27:19.875+0000 7f1375bf17c0 -1 Falling back to public interface 2024-02-10T06:27:20.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:20 smithi139 ceph-mon[87870]: pgmap v69: 161 pgs: 6 peering, 21 active+undersized+degraded, 34 active+undersized, 100 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%) 2024-02-10T06:27:20.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:20 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:20.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:20 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:20.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:20 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:20.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:20 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:20.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:20 smithi139 ceph-mon[85480]: pgmap v69: 161 pgs: 6 peering, 21 active+undersized+degraded, 34 active+undersized, 100 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%) 2024-02-10T06:27:20.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:20 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:20.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:20 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:20.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:20 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:20.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:20 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:20.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:20 smithi150 ceph-mon[68055]: pgmap v69: 161 pgs: 6 peering, 21 active+undersized+degraded, 34 active+undersized, 100 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%) 2024-02-10T06:27:20.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:20 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:20.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:20 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:20.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:20 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:20.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:20 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:21.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:21 smithi139 ceph-mon[85480]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-02-10T06:27:21.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:21 smithi139 ceph-mon[87870]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-02-10T06:27:21.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:21 smithi150 ceph-mon[68055]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-02-10T06:27:22.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:22 smithi139 ceph-mon[85480]: pgmap v70: 161 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 98 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%); 3/627 objects misplaced (0.478%) 2024-02-10T06:27:22.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:22 smithi139 ceph-mon[87870]: pgmap v70: 161 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 98 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%); 3/627 objects misplaced (0.478%) 2024-02-10T06:27:22.837 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[95528]: 2024-02-10T06:27:22.583+0000 7f1375bf17c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-02-10T06:27:22.837 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[95528]: 2024-02-10T06:27:22.613+0000 7f1375bf17c0 -1 osd.0 111 log_to_monitors true 2024-02-10T06:27:22.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:22 smithi150 ceph-mon[68055]: pgmap v70: 161 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 98 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%); 3/627 objects misplaced (0.478%) 2024-02-10T06:27:23.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:23 smithi139 ceph-mon[85480]: from='osd.0 [v2:172.21.15.139:6802/1028295475,v1:172.21.15.139:6803/1028295475]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:27:23.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:23 smithi139 ceph-mon[85480]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:27:23.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:23 smithi139 ceph-mon[85480]: pgmap v71: 161 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 98 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%); 3/627 objects misplaced (0.478%) 2024-02-10T06:27:23.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:23 smithi139 ceph-mon[87870]: from='osd.0 [v2:172.21.15.139:6802/1028295475,v1:172.21.15.139:6803/1028295475]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:27:23.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:23 smithi139 ceph-mon[87870]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:27:23.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:23 smithi139 ceph-mon[87870]: pgmap v71: 161 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 98 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%); 3/627 objects misplaced (0.478%) 2024-02-10T06:27:23.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:23 smithi150 ceph-mon[68055]: from='osd.0 [v2:172.21.15.139:6802/1028295475,v1:172.21.15.139:6803/1028295475]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:27:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:23 smithi150 ceph-mon[68055]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:27:23.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:23 smithi150 ceph-mon[68055]: pgmap v71: 161 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 98 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 67/627 objects degraded (10.686%); 3/627 objects misplaced (0.478%) 2024-02-10T06:27:24.336 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:27:23 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[95528]: 2024-02-10T06:27:23.965+0000 7f1363041700 -1 osd.0 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:27:24.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[87870]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T06:27:24.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[87870]: failed to encode map e116 with expected crc 2024-02-10T06:27:24.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[87870]: failed to encode map e116 with expected crc 2024-02-10T06:27:24.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[87870]: osdmap e116: 8 total, 7 up, 8 in 2024-02-10T06:27:24.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[87870]: from='osd.0 [v2:172.21.15.139:6802/1028295475,v1:172.21.15.139:6803/1028295475]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:24.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[87870]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:24.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:24.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:24.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[87870]: from='osd.0 ' entity='osd.0' 2024-02-10T06:27:24.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[85480]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T06:27:24.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[85480]: failed to encode map e116 with expected crc 2024-02-10T06:27:24.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[85480]: failed to encode map e116 with expected crc 2024-02-10T06:27:24.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[85480]: osdmap e116: 8 total, 7 up, 8 in 2024-02-10T06:27:24.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[85480]: from='osd.0 [v2:172.21.15.139:6802/1028295475,v1:172.21.15.139:6803/1028295475]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:24.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[85480]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:24.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:24.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:24.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:24 smithi139 ceph-mon[85480]: from='osd.0 ' entity='osd.0' 2024-02-10T06:27:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:24 smithi150 ceph-mon[68055]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T06:27:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:24 smithi150 ceph-mon[68055]: failed to encode map e116 with expected crc 2024-02-10T06:27:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:24 smithi150 ceph-mon[68055]: failed to encode map e116 with expected crc 2024-02-10T06:27:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:24 smithi150 ceph-mon[68055]: osdmap e116: 8 total, 7 up, 8 in 2024-02-10T06:27:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:24 smithi150 ceph-mon[68055]: from='osd.0 [v2:172.21.15.139:6802/1028295475,v1:172.21.15.139:6803/1028295475]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:24 smithi150 ceph-mon[68055]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:24 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:24 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:24 smithi150 ceph-mon[68055]: from='osd.0 ' entity='osd.0' 2024-02-10T06:27:25.831 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[87870]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:27:25.832 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[87870]: failed to encode map e117 with expected crc 2024-02-10T06:27:25.832 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[87870]: osd.0 [v2:172.21.15.139:6802/1028295475,v1:172.21.15.139:6803/1028295475] boot 2024-02-10T06:27:25.832 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[87870]: osdmap e117: 8 total, 8 up, 8 in 2024-02-10T06:27:25.832 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:27:25.832 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:25.832 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:25.832 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[87870]: pgmap v74: 161 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 98 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s; 67/627 objects degraded (10.686%); 3/627 objects misplaced (0.478%) 2024-02-10T06:27:25.832 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[85480]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:27:25.833 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[85480]: failed to encode map e117 with expected crc 2024-02-10T06:27:25.833 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[85480]: osd.0 [v2:172.21.15.139:6802/1028295475,v1:172.21.15.139:6803/1028295475] boot 2024-02-10T06:27:25.833 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[85480]: osdmap e117: 8 total, 8 up, 8 in 2024-02-10T06:27:25.833 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:27:25.833 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:25.833 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:25.833 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:25 smithi139 ceph-mon[85480]: pgmap v74: 161 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 98 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s; 67/627 objects degraded (10.686%); 3/627 objects misplaced (0.478%) 2024-02-10T06:27:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:25 smithi150 ceph-mon[68055]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:27:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:25 smithi150 ceph-mon[68055]: failed to encode map e117 with expected crc 2024-02-10T06:27:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:25 smithi150 ceph-mon[68055]: osd.0 [v2:172.21.15.139:6802/1028295475,v1:172.21.15.139:6803/1028295475] boot 2024-02-10T06:27:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:25 smithi150 ceph-mon[68055]: osdmap e117: 8 total, 8 up, 8 in 2024-02-10T06:27:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:25 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:27:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:25 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:25 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:25 smithi150 ceph-mon[68055]: pgmap v74: 161 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 98 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s; 67/627 objects degraded (10.686%); 3/627 objects misplaced (0.478%) 2024-02-10T06:27:26.755 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[85480]: failed to encode map e117 with expected crc 2024-02-10T06:27:26.755 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.755 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.755 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[85480]: osdmap e118: 8 total, 8 up, 8 in 2024-02-10T06:27:26.755 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.755 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.755 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.755 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.755 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.755 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.755 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:26.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[87870]: failed to encode map e117 with expected crc 2024-02-10T06:27:26.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[87870]: osdmap e118: 8 total, 8 up, 8 in 2024-02-10T06:27:26.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.756 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:26 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:26.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:26 smithi150 ceph-mon[68055]: failed to encode map e117 with expected crc 2024-02-10T06:27:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:26 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:26 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:26 smithi150 ceph-mon[68055]: osdmap e118: 8 total, 8 up, 8 in 2024-02-10T06:27:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:26 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:26 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:26 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:26 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:26 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:26 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:26 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[87870]: pgmap v77: 161 pgs: 3 active+clean+remapped, 5 peering, 8 active+undersized+degraded, 13 active+undersized, 132 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 29/627 objects degraded (4.625%); 1/627 objects misplaced (0.159%) 2024-02-10T06:27:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T06:27:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:27 smithi139 ceph-mon[85480]: pgmap v77: 161 pgs: 3 active+clean+remapped, 5 peering, 8 active+undersized+degraded, 13 active+undersized, 132 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 29/627 objects degraded (4.625%); 1/627 objects misplaced (0.159%) 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: failed to encode map e118 with expected crc 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: failed to encode map e119 with expected crc 2024-02-10T06:27:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:27 smithi150 ceph-mon[68055]: pgmap v77: 161 pgs: 3 active+clean+remapped, 5 peering, 8 active+undersized+degraded, 13 active+undersized, 132 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 29/627 objects degraded (4.625%); 1/627 objects misplaced (0.159%) 2024-02-10T06:27:28.675 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[85480]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[85480]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[85480]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[85480]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[85480]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[85480]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T06:27:28.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[87870]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[87870]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[87870]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[87870]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[87870]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.677 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-mon[87870]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T06:27:28.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:28 smithi150 ceph-mon[68055]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:28 smithi150 ceph-mon[68055]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:28 smithi150 ceph-mon[68055]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:28 smithi150 ceph-mon[68055]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:28 smithi150 ceph-mon[68055]: failed to encode map e119 with expected crc 2024-02-10T06:27:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:28 smithi150 ceph-mon[68055]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T06:27:29.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:27:28] "GET /metrics HTTP/1.1" 200 38012 "" "Prometheus/2.43.0" 2024-02-10T06:27:29.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:27:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:27:28.682+0000 7ff625973700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:29.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[85480]: pgmap v78: 161 pgs: 5 peering, 156 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 245 KiB/s rd, 248 op/s; 65 B/s, 0 objects/s recovering 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:29.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:29.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-02-10T06:27:29.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:29 smithi139 ceph-mon[87870]: pgmap v78: 161 pgs: 5 peering, 156 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 245 KiB/s rd, 248 op/s; 65 B/s, 0 objects/s recovering 2024-02-10T06:27:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:27:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:29.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:29.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:29.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-02-10T06:27:29.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:29 smithi150 ceph-mon[68055]: pgmap v78: 161 pgs: 5 peering, 156 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 245 KiB/s rd, 248 op/s; 65 B/s, 0 objects/s recovering 2024-02-10T06:27:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:30 smithi139 ceph-mon[87870]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-02-10T06:27:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:30 smithi139 ceph-mon[87870]: Cluster is now healthy 2024-02-10T06:27:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:30 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:30 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:30 smithi139 ceph-mon[85480]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-02-10T06:27:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:30 smithi139 ceph-mon[85480]: Cluster is now healthy 2024-02-10T06:27:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:30 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:30.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:30.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:30 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:30.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:30 smithi150 ceph-mon[68055]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-02-10T06:27:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:30 smithi150 ceph-mon[68055]: Cluster is now healthy 2024-02-10T06:27:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:30 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:30 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:31.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:31 smithi139 ceph-mon[85480]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 224 KiB/s rd, 227 op/s; 70 B/s, 0 objects/s recovering 2024-02-10T06:27:31.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:31 smithi139 ceph-mon[87870]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 224 KiB/s rd, 227 op/s; 70 B/s, 0 objects/s recovering 2024-02-10T06:27:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:31 smithi150 ceph-mon[68055]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 224 KiB/s rd, 227 op/s; 70 B/s, 0 objects/s recovering 2024-02-10T06:27:34.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:34 smithi139 ceph-mon[85480]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 183 op/s; 56 B/s, 0 objects/s recovering 2024-02-10T06:27:34.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:34 smithi139 ceph-mon[87870]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 183 op/s; 56 B/s, 0 objects/s recovering 2024-02-10T06:27:34.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:34 smithi150 ceph-mon[68055]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 183 op/s; 56 B/s, 0 objects/s recovering 2024-02-10T06:27:36.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:36 smithi139 ceph-mon[85480]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 154 KiB/s rd, 155 op/s; 48 B/s, 0 objects/s recovering 2024-02-10T06:27:36.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:36 smithi139 ceph-mon[87870]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 154 KiB/s rd, 155 op/s; 48 B/s, 0 objects/s recovering 2024-02-10T06:27:36.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:36 smithi150 ceph-mon[68055]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 154 KiB/s rd, 155 op/s; 48 B/s, 0 objects/s recovering 2024-02-10T06:27:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:38 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:38 smithi139 ceph-mon[85480]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 140 op/s; 43 B/s, 0 objects/s recovering 2024-02-10T06:27:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:38 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:38 smithi139 ceph-mon[87870]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 140 op/s; 43 B/s, 0 objects/s recovering 2024-02-10T06:27:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:38 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:38 smithi150 ceph-mon[68055]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 140 op/s; 43 B/s, 0 objects/s recovering 2024-02-10T06:27:39.002 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:27:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:27:38] "GET /metrics HTTP/1.1" 200 38034 "" "Prometheus/2.43.0" 2024-02-10T06:27:39.003 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:27:39.642 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:27:39.642 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (4m) 15s ago 26m 17.7M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (4m) 2m ago 26m 84.3M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (4m) 15s ago 25m 58.1M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (5m) 2m ago 31m 446M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (15m) 15s ago 32m 503M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (2m) 15s ago 33m 47.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (3m) 2m ago 31m 35.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (2m) 15s ago 31m 37.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (5m) 15s ago 27m 16.7M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (5m) 2m ago 27m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (20s) 15s ago 30m 12.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (30m) 15s ago 30m 55.7M 2992M 17.2.0 e1d6a67b021e 74c3d3776d23 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (30m) 15s ago 30m 54.9M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (29m) 15s ago 29m 54.8M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (29m) 2m ago 29m 56.1M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:27:39.643 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (28m) 2m ago 28m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:27:39.644 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (28m) 2m ago 28m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:27:39.644 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (28m) 2m ago 28m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:27:39.644 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (4m) 2m ago 27m 59.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:27:39.644 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (26m) 15s ago 26m 77.3M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:27:39.644 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (26m) 2m ago 26m 75.7M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:27:40.029 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:27:40.030 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:27:40.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:27:40.031 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:27:40.031 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-02-10T06:27:40.031 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 6 2024-02-10T06:27:40.031 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:27:40.031 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:27:40.356 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:27:40.356 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:27:40.356 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:27:40.356 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (1 remaining).", 2024-02-10T06:27:40.356 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:27:40.357 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "1/8 daemons upgraded", 2024-02-10T06:27:40.357 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:27:40.357 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:27:40.357 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:27:40.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:40 smithi139 ceph-mon[85480]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:40.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:40 smithi139 ceph-mon[85480]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:40.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:40 smithi139 ceph-mon[85480]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 122 op/s; 37 B/s, 0 objects/s recovering 2024-02-10T06:27:40.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:40 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/342113199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:40.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:40 smithi139 ceph-mon[87870]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:40.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:40 smithi139 ceph-mon[87870]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:40.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:40 smithi139 ceph-mon[87870]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 122 op/s; 37 B/s, 0 objects/s recovering 2024-02-10T06:27:40.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:40 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/342113199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:40.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:40 smithi150 ceph-mon[68055]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:40.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:40 smithi150 ceph-mon[68055]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:40.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:40 smithi150 ceph-mon[68055]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 122 op/s; 37 B/s, 0 objects/s recovering 2024-02-10T06:27:40.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:40 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/342113199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:27:41.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:41 smithi139 ceph-mon[85480]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:41.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:41 smithi139 ceph-mon[85480]: from='client.54306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:41.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:41 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:41.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:41 smithi139 ceph-mon[87870]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:41.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:41 smithi139 ceph-mon[87870]: from='client.54306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:41.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:41 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:41.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:41 smithi150 ceph-mon[68055]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:41 smithi150 ceph-mon[68055]: from='client.54306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:27:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:41 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:42.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:42 smithi139 ceph-mon[85480]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5 B/s, 0 objects/s recovering 2024-02-10T06:27:42.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:42 smithi139 ceph-mon[87870]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5 B/s, 0 objects/s recovering 2024-02-10T06:27:42.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:42 smithi150 ceph-mon[68055]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5 B/s, 0 objects/s recovering 2024-02-10T06:27:44.576 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:44 smithi139 ceph-mon[85480]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:44.577 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:44 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:44.577 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:44 smithi139 ceph-mon[87870]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:44.577 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:44 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:44 smithi150 ceph-mon[68055]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:44 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:45.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:45.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[85480]: Upgrade: osd.1 is safe to restart 2024-02-10T06:27:45.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:45.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:27:45.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:45.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:45.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:45.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[87870]: Upgrade: osd.1 is safe to restart 2024-02-10T06:27:45.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:45.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:27:45.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:45.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:45 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:45.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:45 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:27:45.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:45 smithi150 ceph-mon[68055]: Upgrade: osd.1 is safe to restart 2024-02-10T06:27:45.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:45 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:45.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:45 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:27:45.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:45 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:45.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:45 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:46.762 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:46 smithi139 ceph-mon[85480]: Upgrade: Updating osd.1 2024-02-10T06:27:46.762 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:46 smithi139 ceph-mon[85480]: Deploying daemon osd.1 on smithi139 2024-02-10T06:27:46.762 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:46 smithi139 ceph-mon[85480]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:46.762 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:46 smithi139 ceph-mon[87870]: Upgrade: Updating osd.1 2024-02-10T06:27:46.763 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:46 smithi139 ceph-mon[87870]: Deploying daemon osd.1 on smithi139 2024-02-10T06:27:46.763 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:46 smithi139 ceph-mon[87870]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:46 smithi150 ceph-mon[68055]: Upgrade: Updating osd.1 2024-02-10T06:27:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:46 smithi150 ceph-mon[68055]: Deploying daemon osd.1 on smithi139 2024-02-10T06:27:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:46 smithi150 ceph-mon[68055]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:48.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:48 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:48.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:48 smithi139 ceph-mon[85480]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:48.279 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:48 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:48.279 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:48 smithi139 ceph-mon[87870]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:48.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:48 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:48 smithi150 ceph-mon[68055]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:48.936 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:27:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:27:48] "GET /metrics HTTP/1.1" 200 38032 "" "Prometheus/2.43.0" 2024-02-10T06:27:48.936 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:48 smithi139 systemd[1]: Stopping Ceph osd.1 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:27:49.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:49 smithi139 ceph-mon[87870]: osd.1 marked itself down and dead 2024-02-10T06:27:49.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:49 smithi139 ceph-mon[85480]: osd.1 marked itself down and dead 2024-02-10T06:27:49.337 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[35284]: 2024-02-10T06:27:48.931+0000 7fb953394700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:27:49.337 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[35284]: 2024-02-10T06:27:48.931+0000 7fb953394700 -1 osd.1 119 *** Got signal Terminated *** 2024-02-10T06:27:49.337 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[35284]: 2024-02-10T06:27:48.931+0000 7fb953394700 -1 osd.1 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:27:49.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:49 smithi150 ceph-mon[68055]: osd.1 marked itself down and dead 2024-02-10T06:27:49.708 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:49 smithi139 podman[99842]: 2024-02-10 06:27:49.451198524 +0000 UTC m=+0.654650703 container died 74c3d3776d23ba5e96f11f1a8f9f9dbf821db776c0c4d7f90fd1f54c0f1f9000 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1, ceph=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, distribution-scope=public, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., vcs-type=git, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-02-10T06:27:49.992 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:49 smithi139 podman[99842]: 2024-02-10 06:27:49.782559639 +0000 UTC m=+0.986011804 container cleanup 74c3d3776d23ba5e96f11f1a8f9f9dbf821db776c0c4d7f90fd1f54c0f1f9000 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_BRANCH=HEAD, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vcs-type=git, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_CLEAN=True, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8) 2024-02-10T06:27:49.992 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:49 smithi139 bash[99842]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1 2024-02-10T06:27:50.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:27:50.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: osdmap e120: 8 total, 7 up, 8 in 2024-02-10T06:27:50.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.276 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:50 smithi150 ceph-mon[68055]: pgmap v89: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:27:50.336 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:49 smithi139 podman[99869]: 2024-02-10 06:27:49.992113292 +0000 UTC m=+0.542025851 container remove 74c3d3776d23ba5e96f11f1a8f9f9dbf821db776c0c4d7f90fd1f54c0f1f9000 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1, architecture=x86_64, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: osdmap e120: 8 total, 7 up, 8 in 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[87870]: pgmap v89: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: osdmap e120: 8 total, 7 up, 8 in 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: failed to encode map e120 with expected crc 2024-02-10T06:27:50.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:50 smithi139 ceph-mon[85480]: pgmap v89: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:27:50.688 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:50 smithi139 podman[99959]: 2024-02-10 06:27:50.303374872 +0000 UTC m=+0.018322616 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:27:50.688 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:50 smithi139 podman[99959]: 2024-02-10 06:27:50.434978431 +0000 UTC m=+0.149926160 container create a05ab916218b165deb3d86e03c5fbd3aadcb6169e0469d21d486fef6297e3911 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True, org.label-schema.license=GPLv2) 2024-02-10T06:27:50.991 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:50 smithi139 podman[99959]: 2024-02-10 06:27:50.687365552 +0000 UTC m=+0.402313282 container init a05ab916218b165deb3d86e03c5fbd3aadcb6169e0469d21d486fef6297e3911 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:27:50.992 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:50 smithi139 podman[99959]: 2024-02-10 06:27:50.690885359 +0000 UTC m=+0.405833098 container start a05ab916218b165deb3d86e03c5fbd3aadcb6169e0469d21d486fef6297e3911 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-10T06:27:50.992 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:50 smithi139 podman[99959]: 2024-02-10 06:27:50.770693613 +0000 UTC m=+0.485641344 container attach a05ab916218b165deb3d86e03c5fbd3aadcb6169e0469d21d486fef6297e3911 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, RELEASE=main-290212a, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:27:51.274 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:50 smithi139 podman[99959]: 2024-02-10 06:27:50.991732266 +0000 UTC m=+0.706680010 container died a05ab916218b165deb3d86e03c5fbd3aadcb6169e0469d21d486fef6297e3911 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[85480]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[85480]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[85480]: osdmap e121: 8 total, 7 up, 8 in 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[85480]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[85480]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[85480]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[85480]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[85480]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[87870]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[87870]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[87870]: osdmap e121: 8 total, 7 up, 8 in 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[87870]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.275 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[87870]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.276 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[87870]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.276 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[87870]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.276 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:51 smithi139 ceph-mon[87870]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:51 smithi150 ceph-mon[68055]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:51 smithi150 ceph-mon[68055]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:51 smithi150 ceph-mon[68055]: osdmap e121: 8 total, 7 up, 8 in 2024-02-10T06:27:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:51 smithi150 ceph-mon[68055]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:51 smithi150 ceph-mon[68055]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:51 smithi150 ceph-mon[68055]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:51 smithi150 ceph-mon[68055]: failed to encode map e121 with expected crc 2024-02-10T06:27:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:51 smithi150 ceph-mon[68055]: failed to encode map e121 with expected crc 2024-02-10T06:27:52.028 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:51 smithi139 podman[99985]: 2024-02-10 06:27:51.750024677 +0000 UTC m=+0.753208740 container remove a05ab916218b165deb3d86e03c5fbd3aadcb6169e0469d21d486fef6297e3911 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-10T06:27:52.029 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:51 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-10T06:27:52.029 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:51 smithi139 systemd[1]: Stopped Ceph osd.1 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:27:52.029 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:51 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.1.service: Consumed 14.643s CPU time. 2024-02-10T06:27:52.029 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:51 smithi139 systemd[1]: Starting Ceph osd.1 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:27:52.298 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:52 smithi139 ceph-mon[85480]: pgmap v91: 161 pgs: 19 active+undersized+degraded, 3 stale+active+clean, 36 active+undersized, 103 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:27:52.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:52 smithi139 ceph-mon[87870]: pgmap v91: 161 pgs: 19 active+undersized+degraded, 3 stale+active+clean, 36 active+undersized, 103 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:27:52.299 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 podman[100092]: 2024-02-10 06:27:52.20102188 +0000 UTC m=+0.109308353 container create a2310a4f21dc25ecbe18b58437d369d692fc82d9a91a00b4012de41e23148f91 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-10T06:27:52.299 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 podman[100092]: 2024-02-10 06:27:52.10978135 +0000 UTC m=+0.018067836 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:27:52.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:52 smithi150 ceph-mon[68055]: pgmap v91: 161 pgs: 19 active+undersized+degraded, 3 stale+active+clean, 36 active+undersized, 103 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:27:52.586 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 podman[100092]: 2024-02-10 06:27:52.453698877 +0000 UTC m=+0.361985350 container init a2310a4f21dc25ecbe18b58437d369d692fc82d9a91a00b4012de41e23148f91 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-10T06:27:52.586 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 podman[100092]: 2024-02-10 06:27:52.456910548 +0000 UTC m=+0.365197031 container start a2310a4f21dc25ecbe18b58437d369d692fc82d9a91a00b4012de41e23148f91 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T06:27:52.586 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 podman[100092]: 2024-02-10 06:27:52.536931124 +0000 UTC m=+0.445217606 container attach a2310a4f21dc25ecbe18b58437d369d692fc82d9a91a00b4012de41e23148f91 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:27:53.279 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:53 smithi139 ceph-mon[85480]: Health check failed: Degraded data redundancy: 78/627 objects degraded (12.440%), 19 pgs degraded (PG_DEGRADED) 2024-02-10T06:27:53.279 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:53 smithi139 ceph-mon[87870]: Health check failed: Degraded data redundancy: 78/627 objects degraded (12.440%), 19 pgs degraded (PG_DEGRADED) 2024-02-10T06:27:53.279 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate[100107]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T06:27:53.279 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate[100107]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 bash[100092]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 bash[100092]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 bash[100092]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 bash[100092]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 bash[100092]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 bash[100092]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 bash[100092]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate[100107]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate[100107]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate[100107]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate[100107]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate[100107]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-10T06:27:53.280 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:52 smithi139 podman[100092]: 2024-02-10 06:27:52.923020797 +0000 UTC m=+0.831307270 container died a2310a4f21dc25ecbe18b58437d369d692fc82d9a91a00b4012de41e23148f91 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:27:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:53 smithi150 ceph-mon[68055]: Health check failed: Degraded data redundancy: 78/627 objects degraded (12.440%), 19 pgs degraded (PG_DEGRADED) 2024-02-10T06:27:53.979 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:53 smithi139 podman[100238]: 2024-02-10 06:27:53.729044435 +0000 UTC m=+0.801017420 container remove a2310a4f21dc25ecbe18b58437d369d692fc82d9a91a00b4012de41e23148f91 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:27:53.980 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:53 smithi139 podman[100304]: 2024-02-10 06:27:53.847532621 +0000 UTC m=+0.018015395 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:27:54.241 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:54 smithi139 ceph-mon[85480]: pgmap v92: 161 pgs: 19 active+undersized+degraded, 3 stale+active+clean, 36 active+undersized, 103 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:27:54.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:54 smithi139 ceph-mon[87870]: pgmap v92: 161 pgs: 19 active+undersized+degraded, 3 stale+active+clean, 36 active+undersized, 103 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:27:54.242 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:53 smithi139 podman[100304]: 2024-02-10 06:27:53.979889847 +0000 UTC m=+0.150372621 container create e33be38d0736264c6c736498e9011dc3294c44fbe353c5f8aaa34cf10b21f6f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1, org.label-schema.build-date=20240131, RELEASE=main-290212a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-02-10T06:27:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:54 smithi150 ceph-mon[68055]: pgmap v92: 161 pgs: 19 active+undersized+degraded, 3 stale+active+clean, 36 active+undersized, 103 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:27:54.586 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:54 smithi139 podman[100304]: 2024-02-10 06:27:54.241386599 +0000 UTC m=+0.411869391 container init e33be38d0736264c6c736498e9011dc3294c44fbe353c5f8aaa34cf10b21f6f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_CLEAN=True) 2024-02-10T06:27:54.586 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:54 smithi139 podman[100304]: 2024-02-10 06:27:54.24468409 +0000 UTC m=+0.415166865 container start e33be38d0736264c6c736498e9011dc3294c44fbe353c5f8aaa34cf10b21f6f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-10T06:27:54.587 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:54 smithi139 ceph-osd[100322]: -- 172.21.15.139:0/565185310 <== mon.1 v2:172.21.15.139:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56275d4fb1e0 con 0x56275d4cd400 2024-02-10T06:27:54.587 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:54 smithi139 bash[100304]: e33be38d0736264c6c736498e9011dc3294c44fbe353c5f8aaa34cf10b21f6f2 2024-02-10T06:27:54.587 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:54 smithi139 systemd[1]: Started Ceph osd.1 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:27:55.086 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:54 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[100318]: 2024-02-10T06:27:54.787+0000 7f91e92827c0 -1 Falling back to public interface 2024-02-10T06:27:55.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:55.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:55.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:55.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:55.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:55.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:55.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:55.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:56.524 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:56 smithi139 ceph-mon[85480]: pgmap v93: 161 pgs: 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T06:27:56.525 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:56 smithi139 ceph-mon[87870]: pgmap v93: 161 pgs: 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T06:27:56.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:56 smithi150 ceph-mon[68055]: pgmap v93: 161 pgs: 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T06:27:57.836 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:57 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[100318]: 2024-02-10T06:27:57.479+0000 7f91e92827c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-02-10T06:27:57.836 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:57 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[100318]: 2024-02-10T06:27:57.507+0000 7f91e92827c0 -1 osd.1 119 log_to_monitors true 2024-02-10T06:27:58.305 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:58 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:58.306 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:58 smithi139 ceph-mon[85480]: pgmap v94: 161 pgs: 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T06:27:58.306 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:58 smithi139 ceph-mon[85480]: from='osd.1 [v2:172.21.15.139:6810/1613273676,v1:172.21.15.139:6811/1613273676]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:27:58.306 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:58 smithi139 ceph-mon[85480]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:27:58.306 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:58 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:58.306 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:58 smithi139 ceph-mon[87870]: pgmap v94: 161 pgs: 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T06:27:58.306 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:58 smithi139 ceph-mon[87870]: from='osd.1 [v2:172.21.15.139:6810/1613273676,v1:172.21.15.139:6811/1613273676]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:27:58.306 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:58 smithi139 ceph-mon[87870]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:27:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:58 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:58 smithi150 ceph-mon[68055]: pgmap v94: 161 pgs: 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T06:27:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:58 smithi150 ceph-mon[68055]: from='osd.1 [v2:172.21.15.139:6810/1613273676,v1:172.21.15.139:6811/1613273676]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:27:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:58 smithi150 ceph-mon[68055]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:27:58.586 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:27:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[100318]: 2024-02-10T06:27:58.467+0000 7f91d66d2700 -1 osd.1 119 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:27:59.013 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:27:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:27:58] "GET /metrics HTTP/1.1" 200 38032 "" "Prometheus/2.43.0" 2024-02-10T06:27:59.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[85480]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T06:27:59.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[85480]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[85480]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[85480]: osdmap e122: 8 total, 7 up, 8 in 2024-02-10T06:27:59.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[85480]: from='osd.1 [v2:172.21.15.139:6810/1613273676,v1:172.21.15.139:6811/1613273676]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:59.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[85480]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:59.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[85480]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[85480]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[85480]: from='osd.1 ' entity='osd.1' 2024-02-10T06:27:59.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[87870]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[87870]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[87870]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[87870]: osdmap e122: 8 total, 7 up, 8 in 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[87870]: from='osd.1 [v2:172.21.15.139:6810/1613273676,v1:172.21.15.139:6811/1613273676]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[87870]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[87870]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[87870]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[87870]: from='osd.1 ' entity='osd.1' 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:59.302 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:27:59 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:59 smithi150 ceph-mon[68055]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T06:27:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:59 smithi150 ceph-mon[68055]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:59 smithi150 ceph-mon[68055]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:59 smithi150 ceph-mon[68055]: osdmap e122: 8 total, 7 up, 8 in 2024-02-10T06:27:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:59 smithi150 ceph-mon[68055]: from='osd.1 [v2:172.21.15.139:6810/1613273676,v1:172.21.15.139:6811/1613273676]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:59 smithi150 ceph-mon[68055]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:27:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:59 smithi150 ceph-mon[68055]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:59 smithi150 ceph-mon[68055]: failed to encode map e122 with expected crc 2024-02-10T06:27:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:59 smithi150 ceph-mon[68055]: from='osd.1 ' entity='osd.1' 2024-02-10T06:27:59.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:59 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:27:59.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:27:59 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: failed to encode map e122 with expected crc 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: failed to encode map e122 with expected crc 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: pgmap v96: 161 pgs: 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: osd.1 [v2:172.21.15.139:6810/1613273676,v1:172.21.15.139:6811/1613273676] boot 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:00.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: failed to encode map e122 with expected crc 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: failed to encode map e122 with expected crc 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: pgmap v96: 161 pgs: 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: osd.1 [v2:172.21.15.139:6810/1613273676,v1:172.21.15.139:6811/1613273676] boot 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:00.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:00 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:00.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: failed to encode map e122 with expected crc 2024-02-10T06:28:00.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: failed to encode map e122 with expected crc 2024-02-10T06:28:00.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: pgmap v96: 161 pgs: 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T06:28:00.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:28:00.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: osd.1 [v2:172.21.15.139:6810/1613273676,v1:172.21.15.139:6811/1613273676] boot 2024-02-10T06:28:00.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T06:28:00.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:28:00.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: failed to encode map e123 with expected crc 2024-02-10T06:28:00.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:00.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:00 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:01.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T06:28:01.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:01.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:01.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: failed to encode map e123 with expected crc 2024-02-10T06:28:01.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T06:28:01.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:01.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:01.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:02.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[85480]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[85480]: pgmap v99: 161 pgs: 8 active+undersized+degraded, 21 active+undersized, 132 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36/627 objects degraded (5.742%) 2024-02-10T06:28:02.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[85480]: Health check update: Degraded data redundancy: 86/627 objects degraded (13.716%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T06:28:02.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[87870]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[87870]: pgmap v99: 161 pgs: 8 active+undersized+degraded, 21 active+undersized, 132 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36/627 objects degraded (5.742%) 2024-02-10T06:28:02.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:02 smithi139 ceph-mon[87870]: Health check update: Degraded data redundancy: 86/627 objects degraded (13.716%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T06:28:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:02 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:02 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:02 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:02 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:02 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:02 smithi150 ceph-mon[68055]: failed to encode map e124 with expected crc 2024-02-10T06:28:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:02 smithi150 ceph-mon[68055]: pgmap v99: 161 pgs: 8 active+undersized+degraded, 21 active+undersized, 132 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36/627 objects degraded (5.742%) 2024-02-10T06:28:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:02 smithi150 ceph-mon[68055]: Health check update: Degraded data redundancy: 86/627 objects degraded (13.716%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T06:28:04.488 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: Upgrade: Finalizing container_image settings 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:04.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.491 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: Upgrade: Finalizing container_image settings 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:04.492 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: pgmap v100: 161 pgs: 1 active+undersized+degraded, 6 active+undersized, 154 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 6/627 objects degraded (0.957%) 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:04.493 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.494 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: pgmap v100: 161 pgs: 1 active+undersized+degraded, 6 active+undersized, 154 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 6/627 objects degraded (0.957%) 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:04.495 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:04.496 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:04.496 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:28:04.496 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.496 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.496 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.496 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.496 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.496 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.496 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.496 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: Upgrade: Complete! 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:04.497 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: Upgrade: Complete! 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:04.498 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:04.499 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:04.499 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.499 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:04.499 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:04.499 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:04.499 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:04.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: Upgrade: Finalizing container_image settings 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: pgmap v100: 161 pgs: 1 active+undersized+degraded, 6 active+undersized, 154 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 6/627 objects degraded (0.957%) 2024-02-10T06:28:04.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:04.628 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: Upgrade: Complete! 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:04.629 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:28:04.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:04.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:04.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:04.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:04.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:04.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:04.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:04.630 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:05.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:05 smithi139 ceph-mon[85480]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:28:05.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:05 smithi139 ceph-mon[85480]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/627 objects degraded (0.957%), 1 pg degraded) 2024-02-10T06:28:05.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:05 smithi139 ceph-mon[85480]: Cluster is now healthy 2024-02-10T06:28:05.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:05 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:05.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:05 smithi139 ceph-mon[87870]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:28:05.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:05 smithi139 ceph-mon[87870]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/627 objects degraded (0.957%), 1 pg degraded) 2024-02-10T06:28:05.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:05 smithi139 ceph-mon[87870]: Cluster is now healthy 2024-02-10T06:28:05.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:05 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:05.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:05 smithi150 ceph-mon[68055]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:28:05.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:05 smithi150 ceph-mon[68055]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/627 objects degraded (0.957%), 1 pg degraded) 2024-02-10T06:28:05.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:05 smithi150 ceph-mon[68055]: Cluster is now healthy 2024-02-10T06:28:05.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:05 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:08.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:08 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:08.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:08 smithi139 ceph-mon[85480]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:28:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:08 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:08 smithi139 ceph-mon[87870]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:28:08.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:08 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:08 smithi150 ceph-mon[68055]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:28:09.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:28:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:28:08] "GET /metrics HTTP/1.1" 200 37963 "" "Prometheus/2.43.0" 2024-02-10T06:28:10.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:10 smithi139 ceph-mon[85480]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:28:10.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:10 smithi139 ceph-mon[87870]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:28:10.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:10 smithi150 ceph-mon[68055]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:28:11.438 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:28:11.749 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:11 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:11.749 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:11 smithi139 ceph-mon[85480]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:11.749 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:11 smithi139 ceph-mon[85480]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-02-10T06:28:11.749 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:11 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:11.749 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:11 smithi139 ceph-mon[87870]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:11.749 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:11 smithi139 ceph-mon[87870]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-02-10T06:28:11.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:11 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:11.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:11 smithi150 ceph-mon[68055]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:11.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:11 smithi150 ceph-mon[68055]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-02-10T06:28:13.436 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:28:13.436 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (5m) 14s ago 27m 17.8M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:28:13.436 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (4m) 2m ago 27m 84.3M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:28:13.436 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (5m) 14s ago 26m 60.3M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:28:13.436 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (6m) 2m ago 31m 446M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:28:13.436 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (16m) 14s ago 33m 504M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:28:13.436 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (3m) 14s ago 33m 48.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:28:13.436 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (3m) 2m ago 32m 35.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:28:13.436 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (2m) 14s ago 32m 38.1M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (5m) 14s ago 28m 17.1M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (5m) 2m ago 27m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (54s) 14s ago 31m 50.5M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (19s) 14s ago 31m 11.8M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (30m) 14s ago 30m 55.7M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (30m) 14s ago 30m 55.8M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (29m) 2m ago 29m 56.1M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (29m) 2m ago 29m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (29m) 2m ago 29m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (28m) 2m ago 28m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (5m) 2m ago 28m 59.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (26m) 14s ago 26m 77.6M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:28:13.437 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (26m) 2m ago 26m 75.7M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:28:14.232 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-02-10T06:28:14.804 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:14 smithi139 ceph-mon[85480]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:14.804 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:14 smithi139 ceph-mon[85480]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:14.804 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:14 smithi139 ceph-mon[87870]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:14.804 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:14 smithi139 ceph-mon[87870]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:14.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:14 smithi150 ceph-mon[68055]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:14.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:14 smithi150 ceph-mon[68055]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:16.186 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:28:16.781 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:16 smithi139 ceph-mon[85480]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:16.781 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:16.781 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:16.781 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:16 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/4274784589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:16.781 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:16 smithi139 ceph-mon[87870]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:16.781 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:16.781 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:16 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:16.782 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:16 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/4274784589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:16.803 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 7'"'"'' 2024-02-10T06:28:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:16 smithi150 ceph-mon[68055]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:16 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/4274784589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:18.190 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:18 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:18.190 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:18 smithi139 ceph-mon[87870]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:18.190 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:18 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:18.190 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:18 smithi139 ceph-mon[85480]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:18.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:18 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:18.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:18 smithi150 ceph-mon[68055]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:18.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:28:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:28:18] "GET /metrics HTTP/1.1" 200 38105 "" "Prometheus/2.43.0" 2024-02-10T06:28:20.832 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:20 smithi139 ceph-mon[85480]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:20.833 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:20 smithi139 ceph-mon[85480]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:20.833 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:20 smithi139 ceph-mon[87870]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:20.833 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:20 smithi139 ceph-mon[87870]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:20.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:20 smithi150 ceph-mon[68055]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:20.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:20 smithi150 ceph-mon[68055]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:20.892 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:28:21.502 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade status' 2024-02-10T06:28:22.685 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:22 smithi139 ceph-mon[85480]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:22.685 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:22 smithi139 ceph-mon[87870]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:22.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:22 smithi150 ceph-mon[68055]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:23.336 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:28:23.336 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": null, 2024-02-10T06:28:23.336 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": false, 2024-02-10T06:28:23.336 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "", 2024-02-10T06:28:23.336 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:28:23.336 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": null, 2024-02-10T06:28:23.336 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "", 2024-02-10T06:28:23.336 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:28:23.336 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:28:23.981 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T06:28:24.575 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:24 smithi139 ceph-mon[85480]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:24.575 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:24 smithi139 ceph-mon[85480]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:24.575 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:24 smithi139 ceph-mon[87870]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:24.575 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:24 smithi139 ceph-mon[87870]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:24.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:24 smithi150 ceph-mon[68055]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:24 smithi150 ceph-mon[68055]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:26.098 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_OK 2024-02-10T06:28:26.727 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:26 smithi139 ceph-mon[85480]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:26.727 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:26 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/2703064010' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:28:26.728 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:26 smithi139 ceph-mon[87870]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:26.728 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:26 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/2703064010' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:28:26.729 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1' 2024-02-10T06:28:26.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:26 smithi150 ceph-mon[68055]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:26 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/2703064010' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:28:28.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:28 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:28.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:28 smithi139 ceph-mon[85480]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:28.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:28 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:28.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:28 smithi139 ceph-mon[87870]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:28 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:28.397 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:28 smithi150 ceph-mon[68055]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:28.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:28:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:28:28] "GET /metrics HTTP/1.1" 200 38105 "" "Prometheus/2.43.0" 2024-02-10T06:28:30.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:30 smithi139 ceph-mon[85480]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:30 smithi139 ceph-mon[85480]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:30 smithi139 ceph-mon[87870]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:30 smithi139 ceph-mon[87870]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:30 smithi150 ceph-mon[68055]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:30.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:30 smithi150 ceph-mon[68055]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:30.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:31.003 INFO:teuthology.orchestra.run.smithi139.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:28:31.559 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T06:28:32.126 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[85480]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:28:32.126 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:32.126 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:32.126 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[85480]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:32.126 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:32.127 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:32.127 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:32.127 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[87870]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:28:32.127 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:32.127 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:32.127 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[87870]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:32.127 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:32.127 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:32.127 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:32 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:32 smithi150 ceph-mon[68055]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:28:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:32 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:32 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:32 smithi150 ceph-mon[68055]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:32 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:32 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:32.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:32 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:33.322 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:33 smithi139 ceph-mon[85480]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:28:33.322 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:33 smithi139 ceph-mon[87870]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:28:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:33 smithi150 ceph-mon[68055]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:28:33.799 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:28:34.246 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:34 smithi139 ceph-mon[85480]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:34.246 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:34 smithi139 ceph-mon[87870]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:34 smithi150 ceph-mon[68055]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:28:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (5m) 35s ago 27m 17.8M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:28:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (5m) 3m ago 27m 84.3M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:28:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (5m) 35s ago 26m 60.3M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:28:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (6m) 3m ago 32m 446M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:28:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (16m) 35s ago 33m 504M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:28:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (3m) 35s ago 33m 48.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:28:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (4m) 3m ago 32m 35.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:28:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (3m) 35s ago 32m 38.1M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:28:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (6m) 35s ago 28m 17.1M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:28:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (6m) 3m ago 28m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:28:34.462 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (75s) 35s ago 31m 50.5M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:28:34.462 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (40s) 35s ago 31m 11.8M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:28:34.462 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (31m) 35s ago 30m 55.7M 2992M 17.2.0 e1d6a67b021e 9d0b338916db 2024-02-10T06:28:34.462 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (30m) 35s ago 30m 55.8M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:28:34.462 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (30m) 3m ago 30m 56.1M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:28:34.462 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (29m) 3m ago 29m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:28:34.462 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (29m) 3m ago 29m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:28:34.462 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (29m) 3m ago 29m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:28:34.462 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (5m) 3m ago 28m 59.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:28:34.462 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (26m) 35s ago 26m 77.6M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:28:34.462 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (26m) 3m ago 26m 75.7M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:28:34.875 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:28:34.875 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:28:34.875 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:28:34.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:28:34.875 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 7 2024-02-10T06:28:34.876 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:28:34.877 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:28:35.225 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:28:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:28:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:28:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) crash,osd. Upgrade limited to 1 daemons (1 remaining).", 2024-02-10T06:28:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:28:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "2/8 daemons upgraded", 2024-02-10T06:28:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:28:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:28:35.226 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:28:35.226 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:35.226 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='client.54366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:35.226 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='client.64306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:35.226 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.226 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:35.226 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:35.226 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.226 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/2216479593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='client.54366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='client.64306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:35.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.228 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:35.228 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.228 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:35.228 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.228 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:28:35.228 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:35 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/2216479593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='client.54366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='client.64306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:28:35.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:35 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/2216479593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:36.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[87870]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[87870]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mgr 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mon 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all crash 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[87870]: Upgrade: osd.2 is safe to restart 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[87870]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[87870]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[85480]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[85480]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mgr 2024-02-10T06:28:36.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mon 2024-02-10T06:28:36.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all crash 2024-02-10T06:28:36.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:28:36.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[85480]: Upgrade: osd.2 is safe to restart 2024-02-10T06:28:36.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[85480]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:36.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:36 smithi139 ceph-mon[85480]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:36 smithi150 ceph-mon[68055]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:28:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:36 smithi150 ceph-mon[68055]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:28:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:36 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mgr 2024-02-10T06:28:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:36 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mon 2024-02-10T06:28:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:36 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all crash 2024-02-10T06:28:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:36 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:28:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:36 smithi150 ceph-mon[68055]: Upgrade: osd.2 is safe to restart 2024-02-10T06:28:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:36 smithi150 ceph-mon[68055]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:28:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:36 smithi150 ceph-mon[68055]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:37.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:37 smithi139 ceph-mon[85480]: Upgrade: Updating osd.2 2024-02-10T06:28:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:37 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:37 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:28:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:37 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:37.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:37 smithi139 ceph-mon[85480]: Deploying daemon osd.2 on smithi139 2024-02-10T06:28:37.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:37 smithi139 ceph-mon[87870]: Upgrade: Updating osd.2 2024-02-10T06:28:37.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:37 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:37.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:37 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:28:37.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:37 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:37.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:37 smithi139 ceph-mon[87870]: Deploying daemon osd.2 on smithi139 2024-02-10T06:28:37.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:37 smithi150 ceph-mon[68055]: Upgrade: Updating osd.2 2024-02-10T06:28:37.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:37 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:37.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:37 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:28:37.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:37 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:37.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:37 smithi150 ceph-mon[68055]: Deploying daemon osd.2 on smithi139 2024-02-10T06:28:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:38 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:38 smithi139 ceph-mon[85480]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:38 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:38 smithi139 ceph-mon[87870]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:38.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:38 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:38.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:38 smithi150 ceph-mon[68055]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:39.007 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:28:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:28:38] "GET /metrics HTTP/1.1" 200 38104 "" "Prometheus/2.43.0" 2024-02-10T06:28:39.840 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:39 smithi139 systemd[1]: Stopping Ceph osd.2 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:28:39.841 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:39 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[38210]: 2024-02-10T06:28:39.619+0000 7f3c6bafc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:28:39.841 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:39 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[38210]: 2024-02-10T06:28:39.619+0000 7f3c6bafc700 -1 osd.2 124 *** Got signal Terminated *** 2024-02-10T06:28:39.841 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:39 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[38210]: 2024-02-10T06:28:39.619+0000 7f3c6bafc700 -1 osd.2 124 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:28:40.835 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:40 smithi139 ceph-mon[85480]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:40.835 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:40 smithi139 ceph-mon[85480]: osd.2 marked itself down and dead 2024-02-10T06:28:40.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:40 smithi139 ceph-mon[87870]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:40.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:40 smithi139 ceph-mon[87870]: osd.2 marked itself down and dead 2024-02-10T06:28:40.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:40 smithi150 ceph-mon[68055]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:40.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:40 smithi150 ceph-mon[68055]: osd.2 marked itself down and dead 2024-02-10T06:28:41.087 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:40 smithi139 podman[106071]: 2024-02-10 06:28:40.83530509 +0000 UTC m=+1.318260454 container stop 9d0b338916dbfc31f7b35cb6bc9d3d91dd095d1f2bcb1c28a0ccc76a4e1f6074 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, version=8, ceph=True, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream) 2024-02-10T06:28:41.087 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:40 smithi139 podman[106071]: 2024-02-10 06:28:40.862996925 +0000 UTC m=+1.345952303 container died 9d0b338916dbfc31f7b35cb6bc9d3d91dd095d1f2bcb1c28a0ccc76a4e1f6074 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vcs-type=git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc.) 2024-02-10T06:28:41.395 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:41 smithi139 podman[106071]: 2024-02-10 06:28:41.211094148 +0000 UTC m=+1.694049511 container cleanup 9d0b338916dbfc31f7b35cb6bc9d3d91dd095d1f2bcb1c28a0ccc76a4e1f6074 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, version=8, name=centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754) 2024-02-10T06:28:41.395 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:41 smithi139 bash[106071]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2 2024-02-10T06:28:41.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:41 smithi139 ceph-mon[87870]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:28:41.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:41 smithi139 ceph-mon[87870]: failed to encode map e125 with expected crc 2024-02-10T06:28:41.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:41 smithi139 ceph-mon[87870]: failed to encode map e125 with expected crc 2024-02-10T06:28:41.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:41 smithi139 ceph-mon[87870]: failed to encode map e125 with expected crc 2024-02-10T06:28:41.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:41 smithi139 ceph-mon[87870]: osdmap e125: 8 total, 7 up, 8 in 2024-02-10T06:28:41.837 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:41 smithi139 podman[106089]: 2024-02-10 06:28:41.395260094 +0000 UTC m=+0.552455765 container remove 9d0b338916dbfc31f7b35cb6bc9d3d91dd095d1f2bcb1c28a0ccc76a4e1f6074 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-type=git, version=8, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, release=754, io.openshift.expose-services=, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, architecture=x86_64, name=centos-stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc.) 2024-02-10T06:28:41.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:41 smithi139 ceph-mon[85480]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:28:41.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:41 smithi139 ceph-mon[85480]: failed to encode map e125 with expected crc 2024-02-10T06:28:41.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:41 smithi139 ceph-mon[85480]: failed to encode map e125 with expected crc 2024-02-10T06:28:41.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:41 smithi139 ceph-mon[85480]: failed to encode map e125 with expected crc 2024-02-10T06:28:41.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:41 smithi139 ceph-mon[85480]: osdmap e125: 8 total, 7 up, 8 in 2024-02-10T06:28:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:41 smithi150 ceph-mon[68055]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:28:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:41 smithi150 ceph-mon[68055]: failed to encode map e125 with expected crc 2024-02-10T06:28:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:41 smithi150 ceph-mon[68055]: failed to encode map e125 with expected crc 2024-02-10T06:28:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:41 smithi150 ceph-mon[68055]: failed to encode map e125 with expected crc 2024-02-10T06:28:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:41 smithi150 ceph-mon[68055]: osdmap e125: 8 total, 7 up, 8 in 2024-02-10T06:28:42.094 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:41 smithi139 podman[106185]: 2024-02-10 06:28:41.745473327 +0000 UTC m=+0.019073246 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:28:42.094 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:41 smithi139 podman[106185]: 2024-02-10 06:28:41.863015774 +0000 UTC m=+0.136615677 container create 6c16a3f0dcf8f220a3095428aad8864c52d073e67a30bda3cdd53799b3fa934c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-02-10T06:28:42.094 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:42 smithi139 podman[106185]: 2024-02-10 06:28:42.090816954 +0000 UTC m=+0.364416858 container init 6c16a3f0dcf8f220a3095428aad8864c52d073e67a30bda3cdd53799b3fa934c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131) 2024-02-10T06:28:42.094 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:42 smithi139 podman[106185]: 2024-02-10 06:28:42.094197593 +0000 UTC m=+0.367797497 container start 6c16a3f0dcf8f220a3095428aad8864c52d073e67a30bda3cdd53799b3fa934c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True) 2024-02-10T06:28:42.390 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:42 smithi139 podman[106185]: 2024-02-10 06:28:42.157442374 +0000 UTC m=+0.431042281 container attach 6c16a3f0dcf8f220a3095428aad8864c52d073e67a30bda3cdd53799b3fa934c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a) 2024-02-10T06:28:42.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: pgmap v120: 161 pgs: 32 peering, 5 stale+active+clean, 124 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:28:42.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: Health check failed: Reduced data availability: 6 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-10T06:28:42.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: osdmap e126: 8 total, 7 up, 8 in 2024-02-10T06:28:42.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.737 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.737 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[85480]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.737 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: pgmap v120: 161 pgs: 32 peering, 5 stale+active+clean, 124 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:28:42.737 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: Health check failed: Reduced data availability: 6 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-10T06:28:42.737 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.737 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: osdmap e126: 8 total, 7 up, 8 in 2024-02-10T06:28:42.737 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.737 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.737 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.737 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.738 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.738 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.738 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.738 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:42 smithi139 ceph-mon[87870]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.738 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:42 smithi139 podman[106185]: 2024-02-10 06:28:42.390004765 +0000 UTC m=+0.663604668 container died 6c16a3f0dcf8f220a3095428aad8864c52d073e67a30bda3cdd53799b3fa934c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-10T06:28:42.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: pgmap v120: 161 pgs: 32 peering, 5 stale+active+clean, 124 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:28:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: Health check failed: Reduced data availability: 6 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-10T06:28:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: osdmap e126: 8 total, 7 up, 8 in 2024-02-10T06:28:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: failed to encode map e126 with expected crc 2024-02-10T06:28:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:42 smithi150 ceph-mon[68055]: failed to encode map e126 with expected crc 2024-02-10T06:28:43.336 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:43 smithi139 podman[106211]: 2024-02-10 06:28:43.003034405 +0000 UTC m=+0.607707190 container remove 6c16a3f0dcf8f220a3095428aad8864c52d073e67a30bda3cdd53799b3fa934c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2) 2024-02-10T06:28:43.337 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:43 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-10T06:28:43.337 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:43 smithi139 systemd[1]: Stopped Ceph osd.2 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:28:43.337 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:43 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.2.service: Consumed 12.254s CPU time. 2024-02-10T06:28:43.337 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:43 smithi139 systemd[1]: Starting Ceph osd.2 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:28:43.724 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:43 smithi139 ceph-mon[85480]: pgmap v122: 161 pgs: 2 active+undersized+degraded, 32 peering, 3 stale+active+clean, 5 active+undersized, 119 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 19/627 objects degraded (3.030%) 2024-02-10T06:28:43.724 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:43 smithi139 ceph-mon[87870]: pgmap v122: 161 pgs: 2 active+undersized+degraded, 32 peering, 3 stale+active+clean, 5 active+undersized, 119 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 19/627 objects degraded (3.030%) 2024-02-10T06:28:43.724 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:43 smithi139 podman[106314]: 2024-02-10 06:28:43.339775953 +0000 UTC m=+0.018197546 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:28:43.724 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:43 smithi139 podman[106314]: 2024-02-10 06:28:43.445692669 +0000 UTC m=+0.124114258 container create a0c54ede9c079e3fddd7725bc542397dc609f5b70d9b8ed72e5cd22d1ad6fd09 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-10T06:28:43.724 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:43 smithi139 podman[106314]: 2024-02-10 06:28:43.723836113 +0000 UTC m=+0.402257704 container init a0c54ede9c079e3fddd7725bc542397dc609f5b70d9b8ed72e5cd22d1ad6fd09 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-10T06:28:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:43 smithi150 ceph-mon[68055]: pgmap v122: 161 pgs: 2 active+undersized+degraded, 32 peering, 3 stale+active+clean, 5 active+undersized, 119 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 19/627 objects degraded (3.030%) 2024-02-10T06:28:44.086 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:43 smithi139 podman[106314]: 2024-02-10 06:28:43.727769342 +0000 UTC m=+0.406190932 container start a0c54ede9c079e3fddd7725bc542397dc609f5b70d9b8ed72e5cd22d1ad6fd09 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2) 2024-02-10T06:28:44.086 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:43 smithi139 podman[106314]: 2024-02-10 06:28:43.798608837 +0000 UTC m=+0.477030418 container attach a0c54ede9c079e3fddd7725bc542397dc609f5b70d9b8ed72e5cd22d1ad6fd09 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-10T06:28:44.449 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate[106329]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T06:28:44.449 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate[106329]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T06:28:44.449 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 bash[106314]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T06:28:44.449 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 bash[106314]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T06:28:44.449 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 bash[106314]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T06:28:44.449 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 bash[106314]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T06:28:44.449 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 bash[106314]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-10T06:28:44.450 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 bash[106314]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T06:28:44.450 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 bash[106314]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-10T06:28:44.450 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate[106329]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T06:28:44.450 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate[106329]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T06:28:44.450 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate[106329]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-10T06:28:44.450 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate[106329]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T06:28:44.450 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate[106329]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-10T06:28:44.450 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 podman[106314]: 2024-02-10 06:28:44.194771919 +0000 UTC m=+0.873193501 container died a0c54ede9c079e3fddd7725bc542397dc609f5b70d9b8ed72e5cd22d1ad6fd09 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-10T06:28:44.819 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:44 smithi139 ceph-mon[85480]: Health check failed: Degraded data redundancy: 19/627 objects degraded (3.030%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T06:28:44.819 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:44 smithi139 ceph-mon[87870]: Health check failed: Degraded data redundancy: 19/627 objects degraded (3.030%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T06:28:44.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:44 smithi150 ceph-mon[68055]: Health check failed: Degraded data redundancy: 19/627 objects degraded (3.030%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T06:28:45.086 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:44 smithi139 podman[106464]: 2024-02-10 06:28:44.819229287 +0000 UTC m=+0.619276478 container remove a0c54ede9c079e3fddd7725bc542397dc609f5b70d9b8ed72e5cd22d1ad6fd09 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-activate, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2) 2024-02-10T06:28:45.087 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:45 smithi139 podman[106530]: 2024-02-10 06:28:45.028369287 +0000 UTC m=+0.112024483 container create 8d3bec1b004008633967f9d0bd957d16277598719975647837c3024b091c68e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131) 2024-02-10T06:28:45.087 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:45 smithi139 podman[106530]: 2024-02-10 06:28:44.933799186 +0000 UTC m=+0.017454393 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:28:45.586 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:45 smithi139 podman[106530]: 2024-02-10 06:28:45.263684884 +0000 UTC m=+0.347340078 container init 8d3bec1b004008633967f9d0bd957d16277598719975647837c3024b091c68e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-10T06:28:45.587 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:45 smithi139 podman[106530]: 2024-02-10 06:28:45.266715263 +0000 UTC m=+0.350370467 container start 8d3bec1b004008633967f9d0bd957d16277598719975647837c3024b091c68e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-10T06:28:45.587 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:45 smithi139 ceph-osd[106548]: -- 172.21.15.139:0/3196399714 <== mon.1 v2:172.21.15.139:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558f1a8cb1e0 con 0x558f1a89d000 2024-02-10T06:28:45.587 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:45 smithi139 bash[106530]: 8d3bec1b004008633967f9d0bd957d16277598719975647837c3024b091c68e4 2024-02-10T06:28:45.587 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:45 smithi139 systemd[1]: Started Ceph osd.2 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:28:46.086 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:45 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[106544]: 2024-02-10T06:28:45.811+0000 7f169f8e27c0 -1 Falling back to public interface 2024-02-10T06:28:46.565 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.566 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.566 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[85480]: pgmap v123: 161 pgs: 3 active+undersized+degraded, 32 peering, 13 active+undersized, 113 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 24/627 objects degraded (3.828%) 2024-02-10T06:28:46.566 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.566 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:46.566 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.566 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:46.566 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.567 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.567 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[87870]: pgmap v123: 161 pgs: 3 active+undersized+degraded, 32 peering, 13 active+undersized, 113 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 24/627 objects degraded (3.828%) 2024-02-10T06:28:46.567 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.567 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:46.567 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.567 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:46 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:46.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:46 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:46 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:46 smithi150 ceph-mon[68055]: pgmap v123: 161 pgs: 3 active+undersized+degraded, 32 peering, 13 active+undersized, 113 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 24/627 objects degraded (3.828%) 2024-02-10T06:28:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:46 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:46 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:46 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:46 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:48.207 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:48 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:48.207 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:48 smithi139 ceph-mon[85480]: pgmap v124: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-02-10T06:28:48.207 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:48 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:48.207 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:48 smithi139 ceph-mon[87870]: pgmap v124: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-02-10T06:28:48.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:48 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:48 smithi150 ceph-mon[68055]: pgmap v124: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-02-10T06:28:48.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:28:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:28:48] "GET /metrics HTTP/1.1" 200 38176 "" "Prometheus/2.43.0" 2024-02-10T06:28:48.864 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[106544]: 2024-02-10T06:28:48.554+0000 7f169f8e27c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-02-10T06:28:48.865 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[106544]: 2024-02-10T06:28:48.578+0000 7f169f8e27c0 -1 osd.2 124 log_to_monitors true 2024-02-10T06:28:49.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:49 smithi139 ceph-mon[85480]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-02-10T06:28:49.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:49 smithi139 ceph-mon[85480]: from='osd.2 [v2:172.21.15.139:6818/3638967059,v1:172.21.15.139:6819/3638967059]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:28:49.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:49 smithi139 ceph-mon[85480]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:28:49.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:49 smithi139 ceph-mon[87870]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-02-10T06:28:49.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:49 smithi139 ceph-mon[87870]: from='osd.2 [v2:172.21.15.139:6818/3638967059,v1:172.21.15.139:6819/3638967059]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:28:49.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:49 smithi139 ceph-mon[87870]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:28:49.337 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:28:49 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[106544]: 2024-02-10T06:28:49.057+0000 7f1690d3a700 -1 osd.2 124 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:28:49.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:49 smithi150 ceph-mon[68055]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-02-10T06:28:49.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:49 smithi150 ceph-mon[68055]: from='osd.2 [v2:172.21.15.139:6818/3638967059,v1:172.21.15.139:6819/3638967059]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:28:49.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:49 smithi150 ceph-mon[68055]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:28:50.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[87870]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T06:28:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[87870]: failed to encode map e127 with expected crc 2024-02-10T06:28:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[87870]: from='osd.2 [v2:172.21.15.139:6818/3638967059,v1:172.21.15.139:6819/3638967059]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:28:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[87870]: failed to encode map e127 with expected crc 2024-02-10T06:28:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[87870]: osdmap e127: 8 total, 7 up, 8 in 2024-02-10T06:28:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[87870]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:28:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[87870]: pgmap v126: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-02-10T06:28:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[87870]: failed to encode map e127 with expected crc 2024-02-10T06:28:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:50.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[85480]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T06:28:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[85480]: failed to encode map e127 with expected crc 2024-02-10T06:28:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[85480]: from='osd.2 [v2:172.21.15.139:6818/3638967059,v1:172.21.15.139:6819/3638967059]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:28:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[85480]: failed to encode map e127 with expected crc 2024-02-10T06:28:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[85480]: osdmap e127: 8 total, 7 up, 8 in 2024-02-10T06:28:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[85480]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:28:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[85480]: pgmap v126: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-02-10T06:28:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[85480]: failed to encode map e127 with expected crc 2024-02-10T06:28:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:50.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:50 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:50 smithi150 ceph-mon[68055]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T06:28:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:50 smithi150 ceph-mon[68055]: failed to encode map e127 with expected crc 2024-02-10T06:28:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:50 smithi150 ceph-mon[68055]: from='osd.2 [v2:172.21.15.139:6818/3638967059,v1:172.21.15.139:6819/3638967059]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:28:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:50 smithi150 ceph-mon[68055]: failed to encode map e127 with expected crc 2024-02-10T06:28:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:50 smithi150 ceph-mon[68055]: osdmap e127: 8 total, 7 up, 8 in 2024-02-10T06:28:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:50 smithi150 ceph-mon[68055]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:28:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:50 smithi150 ceph-mon[68055]: pgmap v126: 161 pgs: 13 active+undersized+degraded, 35 active+undersized, 113 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-02-10T06:28:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:50 smithi150 ceph-mon[68055]: failed to encode map e127 with expected crc 2024-02-10T06:28:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:50 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:50.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:50 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:51.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[85480]: failed to encode map e127 with expected crc 2024-02-10T06:28:51.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[85480]: failed to encode map e127 with expected crc 2024-02-10T06:28:51.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[85480]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:28:51.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[85480]: failed to encode map e128 with expected crc 2024-02-10T06:28:51.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[85480]: failed to encode map e128 with expected crc 2024-02-10T06:28:51.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[85480]: osd.2 [v2:172.21.15.139:6818/3638967059,v1:172.21.15.139:6819/3638967059] boot 2024-02-10T06:28:51.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[85480]: osdmap e128: 8 total, 8 up, 8 in 2024-02-10T06:28:51.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:28:51.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[87870]: failed to encode map e127 with expected crc 2024-02-10T06:28:51.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[87870]: failed to encode map e127 with expected crc 2024-02-10T06:28:51.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[87870]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:28:51.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[87870]: failed to encode map e128 with expected crc 2024-02-10T06:28:51.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[87870]: failed to encode map e128 with expected crc 2024-02-10T06:28:51.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[87870]: osd.2 [v2:172.21.15.139:6818/3638967059,v1:172.21.15.139:6819/3638967059] boot 2024-02-10T06:28:51.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[87870]: osdmap e128: 8 total, 8 up, 8 in 2024-02-10T06:28:51.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:51 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:28:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:51 smithi150 ceph-mon[68055]: failed to encode map e127 with expected crc 2024-02-10T06:28:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:51 smithi150 ceph-mon[68055]: failed to encode map e127 with expected crc 2024-02-10T06:28:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:51 smithi150 ceph-mon[68055]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:28:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:51 smithi150 ceph-mon[68055]: failed to encode map e128 with expected crc 2024-02-10T06:28:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:51 smithi150 ceph-mon[68055]: failed to encode map e128 with expected crc 2024-02-10T06:28:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:51 smithi150 ceph-mon[68055]: osd.2 [v2:172.21.15.139:6818/3638967059,v1:172.21.15.139:6819/3638967059] boot 2024-02-10T06:28:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:51 smithi150 ceph-mon[68055]: osdmap e128: 8 total, 8 up, 8 in 2024-02-10T06:28:51.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:51 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:28:52.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: pgmap v129: 161 pgs: 5 active+undersized+degraded, 19 active+undersized, 137 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[87870]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded (PG_DEGRADED) 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:52.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:52.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: pgmap v129: 161 pgs: 5 active+undersized+degraded, 19 active+undersized, 137 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-02-10T06:28:52.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:52 smithi139 ceph-mon[85480]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded (PG_DEGRADED) 2024-02-10T06:28:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T06:28:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: failed to encode map e129 with expected crc 2024-02-10T06:28:52.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:52.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:52.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: pgmap v129: 161 pgs: 5 active+undersized+degraded, 19 active+undersized, 137 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-02-10T06:28:52.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:52 smithi150 ceph-mon[68055]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded (PG_DEGRADED) 2024-02-10T06:28:54.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:54 smithi139 ceph-mon[85480]: pgmap v130: 161 pgs: 3 active+undersized+degraded, 16 active+undersized, 142 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 7/627 objects degraded (1.116%) 2024-02-10T06:28:54.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:54 smithi139 ceph-mon[87870]: pgmap v130: 161 pgs: 3 active+undersized+degraded, 16 active+undersized, 142 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 7/627 objects degraded (1.116%) 2024-02-10T06:28:54.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:54 smithi150 ceph-mon[68055]: pgmap v130: 161 pgs: 3 active+undersized+degraded, 16 active+undersized, 142 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 7/627 objects degraded (1.116%) 2024-02-10T06:28:56.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:56.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:56.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: Upgrade: Finalizing container_image settings 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:28:56.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:28:56.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:56.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:56.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:28:56.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:56.089 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:56.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: Upgrade: Finalizing container_image settings 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:56.091 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:56.092 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:56.093 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:56.094 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:28:56.095 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.095 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.095 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.095 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.095 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.095 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.095 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.095 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: Upgrade: Complete! 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.096 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 799 B/s rd, 0 op/s 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: Upgrade: Complete! 2024-02-10T06:28:56.097 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 799 B/s rd, 0 op/s 2024-02-10T06:28:56.098 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:55 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:56.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:56.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:56.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: Upgrade: Finalizing container_image settings 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.127 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.128 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:28:56.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: Upgrade: Complete! 2024-02-10T06:28:56.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:56.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:28:56.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:28:56.130 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:56.130 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:56.130 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:56.130 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.130 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:56.130 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:56.130 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:56.130 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.130 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 799 B/s rd, 0 op/s 2024-02-10T06:28:56.130 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:55 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:28:56.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:56 smithi139 ceph-mon[85480]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 3 pgs degraded) 2024-02-10T06:28:56.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:56 smithi139 ceph-mon[85480]: Cluster is now healthy 2024-02-10T06:28:56.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:56 smithi139 ceph-mon[87870]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 3 pgs degraded) 2024-02-10T06:28:56.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:56 smithi139 ceph-mon[87870]: Cluster is now healthy 2024-02-10T06:28:57.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:56 smithi150 ceph-mon[68055]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 3 pgs degraded) 2024-02-10T06:28:57.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:56 smithi150 ceph-mon[68055]: Cluster is now healthy 2024-02-10T06:28:58.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:58 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:28:58 smithi139 ceph-mon[85480]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:28:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:58 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:28:58 smithi139 ceph-mon[87870]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:28:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:58 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:28:58 smithi150 ceph-mon[68055]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:28:59.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:28:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:28:58] "GET /metrics HTTP/1.1" 200 38176 "" "Prometheus/2.43.0" 2024-02-10T06:29:00.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:00 smithi139 ceph-mon[85480]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:29:00.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:00 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:00.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:00 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:00 smithi139 ceph-mon[87870]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:29:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:00 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:00 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:00.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:00 smithi150 ceph-mon[68055]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:29:00.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:00 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:00.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:00 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:01.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:01.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:01 smithi139 ceph-mon[85480]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-02-10T06:29:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:01 smithi139 ceph-mon[87870]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-02-10T06:29:01.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:01.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:01 smithi150 ceph-mon[68055]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-02-10T06:29:04.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:04 smithi139 ceph-mon[85480]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:04.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:04 smithi139 ceph-mon[87870]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:04.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:04 smithi150 ceph-mon[68055]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:06.283 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:29:06.574 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:06 smithi139 ceph-mon[85480]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:06.575 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:06 smithi139 ceph-mon[87870]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:06.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:06 smithi150 ceph-mon[68055]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:07.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:07 smithi139 ceph-mon[85480]: from='client.54396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:07.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:07 smithi139 ceph-mon[87870]: from='client.54396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:07.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:07 smithi150 ceph-mon[68055]: from='client.54396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:08.331 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:29:08.331 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (6m) 18s ago 28m 26.7M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:29:08.331 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (5m) 3m ago 27m 84.3M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (6m) 18s ago 27m 62.5M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (7m) 3m ago 32m 446M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (17m) 18s ago 34m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (4m) 18s ago 34m 51.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (4m) 3m ago 32m 35.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (3m) 18s ago 33m 41.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (6m) 18s ago 28m 17.1M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (6m) 3m ago 28m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (109s) 18s ago 32m 51.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (74s) 18s ago 31m 50.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (23s) 18s ago 31m 11.9M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (31m) 18s ago 31m 56.8M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:29:08.332 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (30m) 3m ago 30m 56.1M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:29:08.333 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (30m) 3m ago 30m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:29:08.333 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (30m) 3m ago 30m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:29:08.333 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (29m) 3m ago 29m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:29:08.333 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (6m) 3m ago 29m 59.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:29:08.333 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (27m) 18s ago 27m 77.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:29:08.333 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (27m) 3m ago 27m 75.7M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:29:08.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:08 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:08.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:08 smithi139 ceph-mon[85480]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:08 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:08.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:08 smithi139 ceph-mon[87870]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:08 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:08.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:08 smithi150 ceph-mon[68055]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:08.938 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:29:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:29:08] "GET /metrics HTTP/1.1" 200 38188 "" "Prometheus/2.43.0" 2024-02-10T06:29:08.962 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-02-10T06:29:09.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:09 smithi139 ceph-mon[85480]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:09.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:09 smithi139 ceph-mon[87870]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:09.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:09 smithi150 ceph-mon[68055]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:10.549 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:10 smithi139 ceph-mon[85480]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:10.549 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:10 smithi139 ceph-mon[87870]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:10.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:10 smithi150 ceph-mon[68055]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:11.087 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:29:11.627 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:11 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/95677820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:11.627 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:11 smithi139 ceph-mon[85480]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:11.627 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:11 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/95677820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:11.627 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:11 smithi139 ceph-mon[87870]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:11.652 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 8'"'"'' 2024-02-10T06:29:11.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:11 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/95677820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:11.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:11 smithi150 ceph-mon[68055]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:14.749 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:14 smithi139 ceph-mon[85480]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:14.749 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:14 smithi139 ceph-mon[87870]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:14.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:14 smithi150 ceph-mon[68055]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:15.638 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:29:15.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:15 smithi139 ceph-mon[85480]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:15.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:15.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:15 smithi139 ceph-mon[87870]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:15.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:15 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:15.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:15 smithi150 ceph-mon[68055]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:15.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:16.206 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade status' 2024-02-10T06:29:16.777 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:16 smithi139 ceph-mon[85480]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:16.778 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:16 smithi139 ceph-mon[87870]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:16 smithi150 ceph-mon[68055]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:18.219 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:29:18.219 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": null, 2024-02-10T06:29:18.219 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": false, 2024-02-10T06:29:18.219 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "", 2024-02-10T06:29:18.220 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:29:18.220 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": null, 2024-02-10T06:29:18.220 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "", 2024-02-10T06:29:18.220 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:29:18.220 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:29:18.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:18 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:18 smithi139 ceph-mon[85480]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:18 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:18 smithi139 ceph-mon[87870]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:18.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:18 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:18.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:18 smithi150 ceph-mon[68055]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:18.914 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T06:29:18.939 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:29:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:29:18] "GET /metrics HTTP/1.1" 200 38182 "" "Prometheus/2.43.0" 2024-02-10T06:29:19.222 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:19 smithi139 ceph-mon[85480]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:19.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:19 smithi139 ceph-mon[87870]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:19 smithi150 ceph-mon[68055]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:20.137 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:20 smithi139 ceph-mon[85480]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:20.138 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:20 smithi139 ceph-mon[87870]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:20.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:20 smithi150 ceph-mon[68055]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:20.902 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_OK 2024-02-10T06:29:21.220 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:21 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/3288827638' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:29:21.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:21 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/3288827638' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:29:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:21 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/3288827638' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:29:21.545 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd' 2024-02-10T06:29:22.198 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:22 smithi139 ceph-mon[85480]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:22.198 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:22 smithi139 ceph-mon[87870]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:22.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:22 smithi150 ceph-mon[68055]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:24.654 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:24 smithi139 ceph-mon[85480]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:24.654 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:24 smithi139 ceph-mon[87870]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:24.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:24 smithi150 ceph-mon[68055]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:25.640 INFO:teuthology.orchestra.run.smithi139.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:29:25.764 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:25 smithi139 ceph-mon[85480]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:25.765 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:25 smithi139 ceph-mon[87870]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:25.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:25 smithi150 ceph-mon[68055]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:26.317 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T06:29:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[85480]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:26.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:29:26.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:26.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[87870]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:26.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:26.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:26.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:26.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:29:26.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:26 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:26 smithi150 ceph-mon[68055]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:26 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:26 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:26 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:26 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:29:26.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:26 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:27.835 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:27 smithi139 ceph-mon[85480]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:29:27.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:27 smithi139 ceph-mon[85480]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:29:27.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:27 smithi139 ceph-mon[87870]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:29:27.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:27 smithi139 ceph-mon[87870]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:29:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:27 smithi150 ceph-mon[68055]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:29:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:27 smithi150 ceph-mon[68055]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:29:28.595 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:29:28.777 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:28 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:28.777 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:28 smithi139 ceph-mon[85480]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:28.778 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:29:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:29:28] "GET /metrics HTTP/1.1" 200 38182 "" "Prometheus/2.43.0" 2024-02-10T06:29:28.778 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:28 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:28.778 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:28 smithi139 ceph-mon[87870]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:28 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:28 smithi150 ceph-mon[68055]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:29.252 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:29:29.252 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (6m) 39s ago 28m 26.7M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:29:29.252 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (6m) 4m ago 28m 84.3M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:29:29.252 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (6m) 39s ago 27m 62.5M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:29:29.252 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (7m) 4m ago 33m 446M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:29:29.252 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (17m) 39s ago 34m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:29:29.252 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (4m) 39s ago 34m 51.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:29:29.252 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (5m) 4m ago 33m 35.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:29:29.252 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (3m) 39s ago 33m 41.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (7m) 39s ago 29m 17.1M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (7m) 4m ago 29m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (2m) 39s ago 32m 51.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (95s) 39s ago 32m 50.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (44s) 39s ago 31m 11.9M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (31m) 39s ago 31m 56.8M 2992M 17.2.0 e1d6a67b021e 6c95425b2ea8 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (31m) 4m ago 31m 56.1M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (30m) 4m ago 30m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (30m) 4m ago 30m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (30m) 4m ago 30m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (6m) 4m ago 29m 59.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (27m) 39s ago 27m 77.9M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:29:29.253 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (27m) 4m ago 27m 75.7M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:29:29.652 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:29:29.653 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:29:29.654 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:29:29.654 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-10T06:29:29.654 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:29:29.654 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:29:29.654 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:29:29.994 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:29:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:29:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:29:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-10T06:29:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:29:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "3/8 daemons upgraded", 2024-02-10T06:29:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:29:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:29:29.995 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:29:30.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='client.54438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:30.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='client.54444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/3548927175' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='client.54438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='client.54444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:29:30.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/3548927175' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.839 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='client.54438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='client.54444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:29:30.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/3548927175' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:30.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:31.772 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:29:31.772 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:29:31.772 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mgr 2024-02-10T06:29:31.772 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mon 2024-02-10T06:29:31.773 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all crash 2024-02-10T06:29:31.773 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:29:31.773 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: Upgrade: osd.3 is safe to restart 2024-02-10T06:29:31.773 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:31.773 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: Upgrade: Updating osd.3 2024-02-10T06:29:31.773 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:31.773 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:29:31.773 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:31.773 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: Deploying daemon osd.3 on smithi139 2024-02-10T06:29:31.773 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[85480]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:31.773 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mgr 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mon 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all crash 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: Upgrade: osd.3 is safe to restart 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: Upgrade: Updating osd.3 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: Deploying daemon osd.3 on smithi139 2024-02-10T06:29:31.774 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:31 smithi139 ceph-mon[87870]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:31.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mgr 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mon 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all crash 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: Upgrade: osd.3 is safe to restart 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: Upgrade: Updating osd.3 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: Deploying daemon osd.3 on smithi139 2024-02-10T06:29:31.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:31 smithi150 ceph-mon[68055]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:34.421 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:34 smithi139 systemd[1]: Stopping Ceph osd.3 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:29:34.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:34 smithi139 ceph-mon[85480]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:34.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:34 smithi139 ceph-mon[85480]: osd.3 marked itself down and dead 2024-02-10T06:29:34.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:34 smithi139 ceph-mon[87870]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:34.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:34 smithi139 ceph-mon[87870]: osd.3 marked itself down and dead 2024-02-10T06:29:34.837 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[41172]: 2024-02-10T06:29:34.419+0000 7f17f073f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:29:34.837 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[41172]: 2024-02-10T06:29:34.419+0000 7f17f073f700 -1 osd.3 129 *** Got signal Terminated *** 2024-02-10T06:29:34.837 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:34 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[41172]: 2024-02-10T06:29:34.419+0000 7f17f073f700 -1 osd.3 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:29:34.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:34 smithi150 ceph-mon[68055]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:34.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:34 smithi150 ceph-mon[68055]: osd.3 marked itself down and dead 2024-02-10T06:29:35.237 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:34 smithi139 podman[112730]: 2024-02-10 06:29:34.982053884 +0000 UTC m=+0.669010380 container died 6c95425b2ea87a8535848f33c1bfd3897f8b3b59e6bd94fa189ee8851e943baa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vendor=Red Hat, Inc., version=8, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, RELEASE=HEAD, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, vcs-type=git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public) 2024-02-10T06:29:35.505 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:35 smithi139 podman[112730]: 2024-02-10 06:29:35.344470122 +0000 UTC m=+1.031426606 container cleanup 6c95425b2ea87a8535848f33c1bfd3897f8b3b59e6bd94fa189ee8851e943baa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, ceph=True, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vcs-type=git, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vendor=Red Hat, Inc., distribution-scope=public, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0) 2024-02-10T06:29:35.505 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:35 smithi139 bash[112730]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3 2024-02-10T06:29:35.836 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:35 smithi139 podman[112747]: 2024-02-10 06:29:35.637494255 +0000 UTC m=+0.651857814 container remove 6c95425b2ea87a8535848f33c1bfd3897f8b3b59e6bd94fa189ee8851e943baa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, release=754, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., vcs-type=git, version=8, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: osdmap e130: 8 total, 7 up, 8 in 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[85480]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:29:35.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: osdmap e130: 8 total, 7 up, 8 in 2024-02-10T06:29:35.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:35 smithi139 ceph-mon[87870]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:29:35.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: osdmap e130: 8 total, 7 up, 8 in 2024-02-10T06:29:35.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: failed to encode map e130 with expected crc 2024-02-10T06:29:35.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:35 smithi150 ceph-mon[68055]: failed to encode map e130 with expected crc 2024-02-10T06:29:36.266 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:35 smithi139 podman[112840]: 2024-02-10 06:29:35.861040578 +0000 UTC m=+0.018974689 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:29:36.266 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:35 smithi139 podman[112840]: 2024-02-10 06:29:35.963786061 +0000 UTC m=+0.121720213 container create 0dc7c45c0ce7c9b3eb3acbe37814cf71614a3cbc2fe8aec87db8185880492da7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, RELEASE=main-290212a, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-10T06:29:36.266 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:36 smithi139 podman[112840]: 2024-02-10 06:29:36.166131002 +0000 UTC m=+0.324065115 container init 0dc7c45c0ce7c9b3eb3acbe37814cf71614a3cbc2fe8aec87db8185880492da7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-02-10T06:29:36.266 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:36 smithi139 podman[112840]: 2024-02-10 06:29:36.170342747 +0000 UTC m=+0.328276869 container start 0dc7c45c0ce7c9b3eb3acbe37814cf71614a3cbc2fe8aec87db8185880492da7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-10T06:29:36.266 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:36 smithi139 podman[112840]: 2024-02-10 06:29:36.266080491 +0000 UTC m=+0.424014616 container attach 0dc7c45c0ce7c9b3eb3acbe37814cf71614a3cbc2fe8aec87db8185880492da7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-10T06:29:36.586 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:36 smithi139 podman[112840]: 2024-02-10 06:29:36.471164511 +0000 UTC m=+0.629098622 container died 0dc7c45c0ce7c9b3eb3acbe37814cf71614a3cbc2fe8aec87db8185880492da7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-02-10T06:29:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[85480]: pgmap v152: 161 pgs: 24 stale+active+clean, 137 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:29:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[85480]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[85480]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[85480]: osdmap e131: 8 total, 7 up, 8 in 2024-02-10T06:29:36.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[85480]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[85480]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[85480]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[85480]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[85480]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[85480]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[85480]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[87870]: pgmap v152: 161 pgs: 24 stale+active+clean, 137 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:29:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[87870]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[87870]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[87870]: osdmap e131: 8 total, 7 up, 8 in 2024-02-10T06:29:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[87870]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[87870]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[87870]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[87870]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[87870]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[87870]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.589 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:36 smithi139 ceph-mon[87870]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:36 smithi150 ceph-mon[68055]: pgmap v152: 161 pgs: 24 stale+active+clean, 137 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:29:36.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:36 smithi150 ceph-mon[68055]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:36 smithi150 ceph-mon[68055]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:36 smithi150 ceph-mon[68055]: osdmap e131: 8 total, 7 up, 8 in 2024-02-10T06:29:36.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:36 smithi150 ceph-mon[68055]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:36 smithi150 ceph-mon[68055]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:36 smithi150 ceph-mon[68055]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:36 smithi150 ceph-mon[68055]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:36 smithi150 ceph-mon[68055]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:36 smithi150 ceph-mon[68055]: failed to encode map e131 with expected crc 2024-02-10T06:29:36.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:36 smithi150 ceph-mon[68055]: failed to encode map e131 with expected crc 2024-02-10T06:29:37.336 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:37 smithi139 podman[112865]: 2024-02-10 06:29:37.119936924 +0000 UTC m=+0.643474364 container remove 0dc7c45c0ce7c9b3eb3acbe37814cf71614a3cbc2fe8aec87db8185880492da7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-10T06:29:37.337 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:37 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-10T06:29:37.337 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:37 smithi139 systemd[1]: Stopped Ceph osd.3 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:29:37.337 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:37 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.3.service: Consumed 16.273s CPU time. 2024-02-10T06:29:37.670 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:37 smithi139 systemd[1]: Starting Ceph osd.3 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:29:37.670 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:37 smithi139 podman[112967]: 2024-02-10 06:29:37.463502135 +0000 UTC m=+0.018929899 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:29:37.670 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:37 smithi139 podman[112967]: 2024-02-10 06:29:37.571389534 +0000 UTC m=+0.126817293 container create 62e45944749c31d96bebed267c867d3d90b589faa5a240dda94a60e5deabe60a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate, RELEASE=main-290212a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:29:38.033 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:37 smithi139 podman[112967]: 2024-02-10 06:29:37.73249946 +0000 UTC m=+0.287927235 container init 62e45944749c31d96bebed267c867d3d90b589faa5a240dda94a60e5deabe60a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-10T06:29:38.033 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:37 smithi139 podman[112967]: 2024-02-10 06:29:37.735814469 +0000 UTC m=+0.291242237 container start 62e45944749c31d96bebed267c867d3d90b589faa5a240dda94a60e5deabe60a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:29:38.033 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:37 smithi139 podman[112967]: 2024-02-10 06:29:37.807052779 +0000 UTC m=+0.362480540 container attach 62e45944749c31d96bebed267c867d3d90b589faa5a240dda94a60e5deabe60a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_BRANCH=HEAD) 2024-02-10T06:29:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-mon[85480]: pgmap v154: 161 pgs: 11 active+undersized+degraded, 12 stale+active+clean, 19 active+undersized, 119 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 37/627 objects degraded (5.901%) 2024-02-10T06:29:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-mon[87870]: pgmap v154: 161 pgs: 11 active+undersized+degraded, 12 stale+active+clean, 19 active+undersized, 119 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 37/627 objects degraded (5.901%) 2024-02-10T06:29:38.337 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate[112981]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T06:29:38.337 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate[112981]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T06:29:38.337 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 bash[112967]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T06:29:38.337 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 bash[112967]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T06:29:38.337 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 bash[112967]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T06:29:38.337 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 bash[112967]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T06:29:38.338 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 bash[112967]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T06:29:38.338 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 bash[112967]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T06:29:38.338 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 bash[112967]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-10T06:29:38.338 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate[112981]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T06:29:38.338 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate[112981]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T06:29:38.338 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate[112981]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T06:29:38.338 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate[112981]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T06:29:38.338 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate[112981]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-10T06:29:38.338 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 podman[112967]: 2024-02-10 06:29:38.200625464 +0000 UTC m=+0.756053217 container died 62e45944749c31d96bebed267c867d3d90b589faa5a240dda94a60e5deabe60a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-10T06:29:38.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:38 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:38 smithi150 ceph-mon[68055]: pgmap v154: 161 pgs: 11 active+undersized+degraded, 12 stale+active+clean, 19 active+undersized, 119 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 37/627 objects degraded (5.901%) 2024-02-10T06:29:38.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:29:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:29:38] "GET /metrics HTTP/1.1" 200 38184 "" "Prometheus/2.43.0" 2024-02-10T06:29:39.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:39 smithi139 ceph-mon[85480]: Health check failed: Degraded data redundancy: 37/627 objects degraded (5.901%), 11 pgs degraded (PG_DEGRADED) 2024-02-10T06:29:39.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:39 smithi139 ceph-mon[87870]: Health check failed: Degraded data redundancy: 37/627 objects degraded (5.901%), 11 pgs degraded (PG_DEGRADED) 2024-02-10T06:29:39.087 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:38 smithi139 podman[113113]: 2024-02-10 06:29:38.835786095 +0000 UTC m=+0.630335062 container remove 62e45944749c31d96bebed267c867d3d90b589faa5a240dda94a60e5deabe60a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:29:39.087 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:39 smithi139 podman[113173]: 2024-02-10 06:29:38.951132062 +0000 UTC m=+0.019517249 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:29:39.087 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:39 smithi139 podman[113173]: 2024-02-10 06:29:39.061595907 +0000 UTC m=+0.129981076 container create e1826d3dcabe858826a6f259fc6005b0ecc8ecc147a86ebee67a4592482087f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-10T06:29:39.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:39 smithi150 ceph-mon[68055]: Health check failed: Degraded data redundancy: 37/627 objects degraded (5.901%), 11 pgs degraded (PG_DEGRADED) 2024-02-10T06:29:39.586 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:39 smithi139 podman[113173]: 2024-02-10 06:29:39.247995658 +0000 UTC m=+0.316380833 container init e1826d3dcabe858826a6f259fc6005b0ecc8ecc147a86ebee67a4592482087f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-10T06:29:39.587 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:39 smithi139 podman[113173]: 2024-02-10 06:29:39.251170958 +0000 UTC m=+0.319556124 container start e1826d3dcabe858826a6f259fc6005b0ecc8ecc147a86ebee67a4592482087f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-10T06:29:39.587 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:39 smithi139 ceph-osd[113193]: -- 172.21.15.139:0/569466801 <== mon.0 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c98256f1e0 con 0x55c982541400 2024-02-10T06:29:39.587 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:39 smithi139 bash[113173]: e1826d3dcabe858826a6f259fc6005b0ecc8ecc147a86ebee67a4592482087f7 2024-02-10T06:29:39.587 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:39 smithi139 systemd[1]: Started Ceph osd.3 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:29:40.086 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:39 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[113189]: 2024-02-10T06:29:39.791+0000 7f05eaf5e7c0 -1 Falling back to public interface 2024-02-10T06:29:40.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:40 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:40.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:40 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:40.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:40 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:40.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:40 smithi139 ceph-mon[85480]: pgmap v155: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-10T06:29:40.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:40 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:40.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:40 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:40.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:40 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:40.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:40 smithi139 ceph-mon[87870]: pgmap v155: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-10T06:29:40.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:40 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:40.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:40 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:40.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:40 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:40.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:40 smithi150 ceph-mon[68055]: pgmap v155: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-10T06:29:42.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:42 smithi139 ceph-mon[85480]: pgmap v156: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-10T06:29:42.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:42 smithi139 ceph-mon[87870]: pgmap v156: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-10T06:29:42.837 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:42 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[113189]: 2024-02-10T06:29:42.486+0000 7f05eaf5e7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-02-10T06:29:42.838 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:42 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[113189]: 2024-02-10T06:29:42.519+0000 7f05eaf5e7c0 -1 osd.3 129 log_to_monitors true 2024-02-10T06:29:42.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:42 smithi150 ceph-mon[68055]: pgmap v156: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-10T06:29:43.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:43 smithi139 ceph-mon[85480]: from='osd.3 [v2:172.21.15.139:6826/247707818,v1:172.21.15.139:6827/247707818]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:29:43.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:43 smithi139 ceph-mon[85480]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:29:43.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:43 smithi139 ceph-mon[87870]: from='osd.3 [v2:172.21.15.139:6826/247707818,v1:172.21.15.139:6827/247707818]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:29:43.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:43 smithi139 ceph-mon[87870]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:29:43.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:43 smithi150 ceph-mon[68055]: from='osd.3 [v2:172.21.15.139:6826/247707818,v1:172.21.15.139:6827/247707818]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:29:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:43 smithi150 ceph-mon[68055]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:29:44.336 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:29:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[113189]: 2024-02-10T06:29:43.991+0000 7f05d83ae700 -1 osd.3 129 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:29:44.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[87870]: pgmap v157: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:29:44.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[87870]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T06:29:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[87870]: failed to encode map e132 with expected crc 2024-02-10T06:29:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[87870]: failed to encode map e132 with expected crc 2024-02-10T06:29:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[87870]: osdmap e132: 8 total, 7 up, 8 in 2024-02-10T06:29:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[87870]: from='osd.3 [v2:172.21.15.139:6826/247707818,v1:172.21.15.139:6827/247707818]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:29:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[87870]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:29:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[87870]: from='osd.3 ' entity='osd.3' 2024-02-10T06:29:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[85480]: pgmap v157: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:29:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[85480]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T06:29:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[85480]: failed to encode map e132 with expected crc 2024-02-10T06:29:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[85480]: failed to encode map e132 with expected crc 2024-02-10T06:29:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[85480]: osdmap e132: 8 total, 7 up, 8 in 2024-02-10T06:29:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[85480]: from='osd.3 [v2:172.21.15.139:6826/247707818,v1:172.21.15.139:6827/247707818]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:29:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[85480]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:29:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:44.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:44 smithi139 ceph-mon[85480]: from='osd.3 ' entity='osd.3' 2024-02-10T06:29:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:44 smithi150 ceph-mon[68055]: pgmap v157: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:29:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:44 smithi150 ceph-mon[68055]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T06:29:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:44 smithi150 ceph-mon[68055]: failed to encode map e132 with expected crc 2024-02-10T06:29:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:44 smithi150 ceph-mon[68055]: failed to encode map e132 with expected crc 2024-02-10T06:29:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:44 smithi150 ceph-mon[68055]: osdmap e132: 8 total, 7 up, 8 in 2024-02-10T06:29:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:44 smithi150 ceph-mon[68055]: from='osd.3 [v2:172.21.15.139:6826/247707818,v1:172.21.15.139:6827/247707818]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:29:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:44 smithi150 ceph-mon[68055]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-02-10T06:29:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:44 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:44 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:44.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:44 smithi150 ceph-mon[68055]: from='osd.3 ' entity='osd.3' 2024-02-10T06:29:46.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: failed to encode map e132 with expected crc 2024-02-10T06:29:46.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:46.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:46.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:29:46.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: osd.3 [v2:172.21.15.139:6826/247707818,v1:172.21.15.139:6827/247707818] boot 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: pgmap v160: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:46.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: failed to encode map e132 with expected crc 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: osd.3 [v2:172.21.15.139:6826/247707818,v1:172.21.15.139:6827/247707818] boot 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: pgmap v160: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-10T06:29:46.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:46.340 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:45 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:46.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: failed to encode map e132 with expected crc 2024-02-10T06:29:46.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:46.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:46.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:29:46.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:29:46.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: osd.3 [v2:172.21.15.139:6826/247707818,v1:172.21.15.139:6827/247707818] boot 2024-02-10T06:29:46.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T06:29:46.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: failed to encode map e133 with expected crc 2024-02-10T06:29:46.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: pgmap v160: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-10T06:29:46.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:46.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:45 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:47.177 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[85480]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.177 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[85480]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.177 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[85480]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T06:29:47.177 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[85480]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.177 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[85480]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.177 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[85480]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.177 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[85480]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.177 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[85480]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.177 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[85480]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.178 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[87870]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.178 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[87870]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.178 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[87870]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T06:29:47.178 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[87870]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.178 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[87870]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.178 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[87870]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.178 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[87870]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.178 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[87870]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.178 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:47 smithi139 ceph-mon[87870]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:47 smithi150 ceph-mon[68055]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:47 smithi150 ceph-mon[68055]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:47 smithi150 ceph-mon[68055]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T06:29:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:47 smithi150 ceph-mon[68055]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:47 smithi150 ceph-mon[68055]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:47 smithi150 ceph-mon[68055]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:47 smithi150 ceph-mon[68055]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:47 smithi150 ceph-mon[68055]: failed to encode map e134 with expected crc 2024-02-10T06:29:47.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:47 smithi150 ceph-mon[68055]: failed to encode map e134 with expected crc 2024-02-10T06:29:48.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:48 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:48.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:48 smithi139 ceph-mon[85480]: Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:29:48.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:48 smithi139 ceph-mon[85480]: pgmap v162: 161 pgs: 16 active+undersized+degraded, 28 active+undersized, 117 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-02-10T06:29:48.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:48 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:48.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:48 smithi139 ceph-mon[87870]: Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:29:48.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:48 smithi139 ceph-mon[87870]: pgmap v162: 161 pgs: 16 active+undersized+degraded, 28 active+undersized, 117 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-02-10T06:29:48.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:48 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:48 smithi150 ceph-mon[68055]: Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:29:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:48 smithi150 ceph-mon[68055]: pgmap v162: 161 pgs: 16 active+undersized+degraded, 28 active+undersized, 117 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-02-10T06:29:48.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:29:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:29:48] "GET /metrics HTTP/1.1" 200 38204 "" "Prometheus/2.43.0" 2024-02-10T06:29:48.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:29:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:29:48.748+0000 7ff625973700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T06:29:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:29:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:29:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:29:49.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T06:29:49.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:49 smithi150 ceph-mon[68055]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:29:50.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T06:29:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[87870]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T06:29:50.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:49 smithi139 ceph-mon[85480]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:29:50.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:50 smithi150 ceph-mon[68055]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 16 pgs degraded) 2024-02-10T06:29:50.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:50 smithi150 ceph-mon[68055]: Cluster is now healthy 2024-02-10T06:29:51.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:50 smithi139 ceph-mon[85480]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 16 pgs degraded) 2024-02-10T06:29:51.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:50 smithi139 ceph-mon[85480]: Cluster is now healthy 2024-02-10T06:29:51.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:50 smithi139 ceph-mon[87870]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 16 pgs degraded) 2024-02-10T06:29:51.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:50 smithi139 ceph-mon[87870]: Cluster is now healthy 2024-02-10T06:29:51.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:51 smithi150 ceph-mon[68055]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:29:52.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:51 smithi139 ceph-mon[85480]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:29:52.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:51 smithi139 ceph-mon[87870]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:29:54.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:54 smithi139 ceph-mon[85480]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-02-10T06:29:54.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:54 smithi139 ceph-mon[87870]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-02-10T06:29:54.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:54 smithi150 ceph-mon[68055]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-02-10T06:29:56.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:56 smithi139 ceph-mon[87870]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:29:56.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:56 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:56.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:56 smithi139 ceph-mon[85480]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:29:56.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:56 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:56.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:56 smithi150 ceph-mon[68055]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:29:56.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:56 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:29:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:58 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:29:58 smithi139 ceph-mon[85480]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-02-10T06:29:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:58 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:29:58 smithi139 ceph-mon[87870]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-02-10T06:29:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:58 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:29:58 smithi150 ceph-mon[68055]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-02-10T06:29:59.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:29:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:29:58] "GET /metrics HTTP/1.1" 200 38204 "" "Prometheus/2.43.0" 2024-02-10T06:30:00.351 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:30:00.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:00 smithi139 ceph-mon[85480]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:00.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:00 smithi139 ceph-mon[85480]: overall HEALTH_OK 2024-02-10T06:30:00.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:00 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:00.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:00 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:00 smithi139 ceph-mon[87870]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:00 smithi139 ceph-mon[87870]: overall HEALTH_OK 2024-02-10T06:30:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:00 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:00 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:00.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:00 smithi150 ceph-mon[68055]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:00.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:00 smithi150 ceph-mon[68055]: overall HEALTH_OK 2024-02-10T06:30:00.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:00 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:00.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:00 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:00.999 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (6m) 17s ago 29m 26.3M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (6m) 4m ago 28m 84.3M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (7m) 17s ago 28m 64.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (8m) 4m ago 33m 446M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (18m) 17s ago 35m 508M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (4m) 17s ago 35m 54.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (5m) 4m ago 33m 35.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (4m) 17s ago 33m 41.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (7m) 17s ago 29m 17.0M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (7m) 4m ago 29m 15.9M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (2m) 17s ago 33m 53.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:30:01.000 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (2m) 17s ago 32m 51.5M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:30:01.001 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (76s) 17s ago 32m 40.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:30:01.001 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (22s) 17s ago 32m 12.6M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e1826d3dcabe 2024-02-10T06:30:01.001 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (31m) 4m ago 31m 56.1M 3248M 17.2.0 e1d6a67b021e 91249b4f1320 2024-02-10T06:30:01.001 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (31m) 4m ago 31m 55.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:30:01.001 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (30m) 4m ago 30m 52.9M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:30:01.001 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (30m) 4m ago 30m 56.3M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:30:01.001 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (7m) 4m ago 29m 59.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:30:01.001 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (28m) 17s ago 28m 78.2M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:30:01.001 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (28m) 4m ago 28m 75.7M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:30:01.389 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:30:01.389 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:30:01.389 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:30:01.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:30:01.389 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:30:01.389 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:30:01.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:30:01.389 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:30:01.389 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-10T06:30:01.389 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T06:30:01.390 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:30:01.390 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:30:01.390 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:30:01.390 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:30:01.390 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:30:01.390 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-10T06:30:01.390 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 9 2024-02-10T06:30:01.390 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:30:01.390 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:30:01.721 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:30:01.721 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:30:01.721 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:30:01.721 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-10T06:30:01.721 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:30:01.721 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "4/8 daemons upgraded", 2024-02-10T06:30:01.721 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:30:01.721 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:30:01.721 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:30:01.722 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:01 smithi139 ceph-mon[85480]: from='client.54474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:01.722 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:01 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/3352186809' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:01.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:01 smithi139 ceph-mon[87870]: from='client.54474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:01.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:01 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/3352186809' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:01.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:01 smithi150 ceph-mon[68055]: from='client.54474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:01.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:01 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/3352186809' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:02.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:02 smithi139 ceph-mon[85480]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:02.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:02 smithi139 ceph-mon[85480]: from='client.64420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:02.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:02 smithi139 ceph-mon[85480]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:02.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:02 smithi139 ceph-mon[85480]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:02.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:02 smithi139 ceph-mon[87870]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:02.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:02 smithi139 ceph-mon[87870]: from='client.64420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:02.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:02 smithi139 ceph-mon[87870]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:02.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:02 smithi139 ceph-mon[87870]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:02.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:02 smithi150 ceph-mon[68055]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:02.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:02 smithi150 ceph-mon[68055]: from='client.64420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:02.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:02 smithi150 ceph-mon[68055]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:02.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:02 smithi150 ceph-mon[68055]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:03.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:03 smithi139 ceph-mon[85480]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:03 smithi139 ceph-mon[87870]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:03.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:03 smithi150 ceph-mon[68055]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:04.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:30:04.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:04 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:30:04.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:04 smithi150 ceph-mon[68055]: Upgrade: osd.4 is safe to restart 2024-02-10T06:30:04.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:30:04.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:04 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:30:04.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:04 smithi139 ceph-mon[87870]: Upgrade: osd.4 is safe to restart 2024-02-10T06:30:04.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:30:04.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:04 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:30:04.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:04 smithi139 ceph-mon[85480]: Upgrade: osd.4 is safe to restart 2024-02-10T06:30:05.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:05 smithi139 ceph-mon[85480]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:05.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:05 smithi139 ceph-mon[87870]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:05.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:05 smithi150 ceph-mon[68055]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:06 smithi150 ceph-mon[68055]: Upgrade: Updating osd.4 2024-02-10T06:30:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:06 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:06 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:30:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:06 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:06 smithi150 ceph-mon[68055]: Deploying daemon osd.4 on smithi150 2024-02-10T06:30:06.891 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:06 smithi139 ceph-mon[87870]: Upgrade: Updating osd.4 2024-02-10T06:30:06.892 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:06 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:06.892 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:06 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:30:06.892 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:06 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:06.892 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:06 smithi139 ceph-mon[87870]: Deploying daemon osd.4 on smithi150 2024-02-10T06:30:06.892 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:06 smithi139 ceph-mon[85480]: Upgrade: Updating osd.4 2024-02-10T06:30:06.892 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:06 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:06.892 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:06 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:30:06.892 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:06 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:06.892 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:06 smithi139 ceph-mon[85480]: Deploying daemon osd.4 on smithi150 2024-02-10T06:30:08.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:08 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:08.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:08 smithi139 ceph-mon[85480]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:08 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:08 smithi139 ceph-mon[87870]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:08.361 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:08 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:08.361 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:08 smithi150 ceph-mon[68055]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:09.000 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:08 smithi150 systemd[1]: Stopping Ceph osd.4 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:30:09.044 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:30:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:30:08] "GET /metrics HTTP/1.1" 200 38192 "" "Prometheus/2.43.0" 2024-02-10T06:30:09.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:09 smithi139 ceph-mon[85480]: osd.4 marked itself down and dead 2024-02-10T06:30:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:09 smithi139 ceph-mon[87870]: osd.4 marked itself down and dead 2024-02-10T06:30:09.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:09 smithi150 ceph-mon[68055]: osd.4 marked itself down and dead 2024-02-10T06:30:09.375 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[29022]: 2024-02-10T06:30:08.998+0000 7f48a8d65700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:30:09.375 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[29022]: 2024-02-10T06:30:08.998+0000 7f48a8d65700 -1 osd.4 134 *** Got signal Terminated *** 2024-02-10T06:30:09.375 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[29022]: 2024-02-10T06:30:08.998+0000 7f48a8d65700 -1 osd.4 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:30:09.708 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:09 smithi150 podman[78351]: 2024-02-10 06:30:09.416316762 +0000 UTC m=+0.514827169 container died 91249b4f1320a2a80b4b99ffdf642122b3640526929a870434d765c5416a295b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, release=754, io.openshift.expose-services=, com.redhat.component=centos-stream-container, ceph=True, architecture=x86_64, maintainer=Guillaume Abrioux , RELEASE=HEAD, vendor=Red Hat, Inc., io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T06:30:10.046 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:09 smithi150 podman[78351]: 2024-02-10 06:30:09.798652554 +0000 UTC m=+0.897162952 container cleanup 91249b4f1320a2a80b4b99ffdf642122b3640526929a870434d765c5416a295b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4, ceph=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, version=8, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, name=centos-stream) 2024-02-10T06:30:10.046 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:09 smithi150 bash[78351]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4 2024-02-10T06:30:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[85480]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:30:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[85480]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[85480]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[85480]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[85480]: osdmap e135: 8 total, 7 up, 8 in 2024-02-10T06:30:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[85480]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[85480]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[85480]: pgmap v174: 161 pgs: 24 stale+active+clean, 137 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:10.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[87870]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:30:10.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[87870]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[87870]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[87870]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[87870]: osdmap e135: 8 total, 7 up, 8 in 2024-02-10T06:30:10.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[87870]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[87870]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:10 smithi139 ceph-mon[87870]: pgmap v174: 161 pgs: 24 stale+active+clean, 137 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:10.374 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:10 smithi150 podman[78378]: 2024-02-10 06:30:10.050481971 +0000 UTC m=+0.632457880 container remove 91249b4f1320a2a80b4b99ffdf642122b3640526929a870434d765c5416a295b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4, name=centos-stream, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, ceph=True, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, release=754, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-10T06:30:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:10 smithi150 ceph-mon[68055]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:30:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:10 smithi150 ceph-mon[68055]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:10 smithi150 ceph-mon[68055]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:10 smithi150 ceph-mon[68055]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:10 smithi150 ceph-mon[68055]: osdmap e135: 8 total, 7 up, 8 in 2024-02-10T06:30:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:10 smithi150 ceph-mon[68055]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:10 smithi150 ceph-mon[68055]: failed to encode map e135 with expected crc 2024-02-10T06:30:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:10 smithi150 ceph-mon[68055]: pgmap v174: 161 pgs: 24 stale+active+clean, 137 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:10.663 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:10 smithi150 podman[78473]: 2024-02-10 06:30:10.293868268 +0000 UTC m=+0.018042408 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:30:10.663 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:10 smithi150 podman[78473]: 2024-02-10 06:30:10.397007554 +0000 UTC m=+0.121181682 container create 6060dfdacc246f0a14d3a49d96fe648d77260469138fc8b072b40db68b2cc5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=) 2024-02-10T06:30:10.957 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:10 smithi150 podman[78473]: 2024-02-10 06:30:10.662958488 +0000 UTC m=+0.387132630 container init 6060dfdacc246f0a14d3a49d96fe648d77260469138fc8b072b40db68b2cc5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux ) 2024-02-10T06:30:10.957 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:10 smithi150 podman[78473]: 2024-02-10 06:30:10.666533775 +0000 UTC m=+0.390707912 container start 6060dfdacc246f0a14d3a49d96fe648d77260469138fc8b072b40db68b2cc5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a) 2024-02-10T06:30:10.957 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:10 smithi150 podman[78473]: 2024-02-10 06:30:10.837630841 +0000 UTC m=+0.561804970 container attach 6060dfdacc246f0a14d3a49d96fe648d77260469138fc8b072b40db68b2cc5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True) 2024-02-10T06:30:11.216 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:10 smithi150 podman[78473]: 2024-02-10 06:30:10.957308911 +0000 UTC m=+0.681483046 container died 6060dfdacc246f0a14d3a49d96fe648d77260469138fc8b072b40db68b2cc5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a) 2024-02-10T06:30:11.216 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:11 smithi150 ceph-mon[68055]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.216 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:11 smithi150 ceph-mon[68055]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.216 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:11 smithi150 ceph-mon[68055]: osdmap e136: 8 total, 7 up, 8 in 2024-02-10T06:30:11.216 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:11 smithi150 ceph-mon[68055]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.216 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:11 smithi150 ceph-mon[68055]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.216 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:11 smithi150 ceph-mon[68055]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[85480]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[85480]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[85480]: osdmap e136: 8 total, 7 up, 8 in 2024-02-10T06:30:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[85480]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[85480]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[85480]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[87870]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[87870]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[87870]: osdmap e136: 8 total, 7 up, 8 in 2024-02-10T06:30:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[87870]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[87870]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:11 smithi139 ceph-mon[87870]: failed to encode map e136 with expected crc 2024-02-10T06:30:11.875 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:11 smithi150 podman[78500]: 2024-02-10 06:30:11.491848137 +0000 UTC m=+0.527792870 container remove 6060dfdacc246f0a14d3a49d96fe648d77260469138fc8b072b40db68b2cc5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T06:30:11.875 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:11 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-10T06:30:11.875 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:11 smithi150 systemd[1]: Stopped Ceph osd.4 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:30:11.875 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:11 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.4.service: Consumed 17.584s CPU time. 2024-02-10T06:30:11.875 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:11 smithi150 systemd[1]: Starting Ceph osd.4 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:30:12.221 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:12 smithi150 ceph-mon[68055]: pgmap v176: 161 pgs: 13 active+undersized+degraded, 16 stale+active+clean, 16 active+undersized, 116 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 59/627 objects degraded (9.410%) 2024-02-10T06:30:12.221 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:11 smithi150 podman[78608]: 2024-02-10 06:30:11.836093107 +0000 UTC m=+0.019313593 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:30:12.222 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:11 smithi150 podman[78608]: 2024-02-10 06:30:11.943116801 +0000 UTC m=+0.126337271 container create 7b9fe301a56ce2e5793a59ede45b2e0f34743b2d2096429c7562998f627483dc (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-290212a, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-10T06:30:12.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:12 smithi139 ceph-mon[85480]: pgmap v176: 161 pgs: 13 active+undersized+degraded, 16 stale+active+clean, 16 active+undersized, 116 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 59/627 objects degraded (9.410%) 2024-02-10T06:30:12.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:12 smithi139 ceph-mon[87870]: pgmap v176: 161 pgs: 13 active+undersized+degraded, 16 stale+active+clean, 16 active+undersized, 116 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 59/627 objects degraded (9.410%) 2024-02-10T06:30:12.624 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 podman[78608]: 2024-02-10 06:30:12.220800138 +0000 UTC m=+0.404020619 container init 7b9fe301a56ce2e5793a59ede45b2e0f34743b2d2096429c7562998f627483dc (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-10T06:30:12.625 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 podman[78608]: 2024-02-10 06:30:12.224546894 +0000 UTC m=+0.407767371 container start 7b9fe301a56ce2e5793a59ede45b2e0f34743b2d2096429c7562998f627483dc (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:30:12.625 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 podman[78608]: 2024-02-10 06:30:12.295658284 +0000 UTC m=+0.478878762 container attach 7b9fe301a56ce2e5793a59ede45b2e0f34743b2d2096429c7562998f627483dc (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a) 2024-02-10T06:30:12.990 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate[78623]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate[78623]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 bash[78608]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 bash[78608]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 bash[78608]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 bash[78608]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 bash[78608]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 bash[78608]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 bash[78608]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate[78623]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate[78623]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate[78623]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-10T06:30:12.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate[78623]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T06:30:12.992 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate[78623]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-10T06:30:12.992 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:12 smithi150 podman[78608]: 2024-02-10 06:30:12.695677098 +0000 UTC m=+0.878897572 container died 7b9fe301a56ce2e5793a59ede45b2e0f34743b2d2096429c7562998f627483dc (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:30:13.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:13 smithi150 ceph-mon[68055]: Health check failed: Degraded data redundancy: 59/627 objects degraded (9.410%), 13 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:13 smithi139 ceph-mon[85480]: Health check failed: Degraded data redundancy: 59/627 objects degraded (9.410%), 13 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:13.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:13 smithi139 ceph-mon[87870]: Health check failed: Degraded data redundancy: 59/627 objects degraded (9.410%), 13 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:13.624 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:13 smithi150 podman[78739]: 2024-02-10 06:30:13.316397879 +0000 UTC m=+0.615516383 container remove 7b9fe301a56ce2e5793a59ede45b2e0f34743b2d2096429c7562998f627483dc (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-10T06:30:13.624 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:13 smithi150 podman[78807]: 2024-02-10 06:30:13.43073213 +0000 UTC m=+0.016914136 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:30:13.625 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:13 smithi150 podman[78807]: 2024-02-10 06:30:13.559128623 +0000 UTC m=+0.145310616 container create f61046f26c977333ff2e22978034a16770c2c24c061346455c3467aa2b137271 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_CLEAN=True) 2024-02-10T06:30:14.070 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:13 smithi150 podman[78807]: 2024-02-10 06:30:13.809632574 +0000 UTC m=+0.395814580 container init f61046f26c977333ff2e22978034a16770c2c24c061346455c3467aa2b137271 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:30:14.070 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:13 smithi150 podman[78807]: 2024-02-10 06:30:13.812808118 +0000 UTC m=+0.398990115 container start f61046f26c977333ff2e22978034a16770c2c24c061346455c3467aa2b137271 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, ceph=True, RELEASE=main-290212a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:30:14.070 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:13 smithi150 ceph-osd[78826]: -- 172.21.15.150:0/2016105187 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bb1143b1e0 con 0x55bb1140d000 2024-02-10T06:30:14.070 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:13 smithi150 bash[78807]: f61046f26c977333ff2e22978034a16770c2c24c061346455c3467aa2b137271 2024-02-10T06:30:14.070 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:13 smithi150 systemd[1]: Started Ceph osd.4 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:30:14.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:14 smithi139 ceph-mon[87870]: pgmap v177: 161 pgs: 17 active+undersized+degraded, 12 stale+active+clean, 24 active+undersized, 108 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 72/627 objects degraded (11.483%) 2024-02-10T06:30:14.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:14 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:14.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:14 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:14.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:14 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:14.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:14 smithi139 ceph-mon[85480]: pgmap v177: 161 pgs: 17 active+undersized+degraded, 12 stale+active+clean, 24 active+undersized, 108 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 72/627 objects degraded (11.483%) 2024-02-10T06:30:14.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:14 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:14.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:14 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:14.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:14 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:14.351 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:14 smithi150 ceph-mon[68055]: pgmap v177: 161 pgs: 17 active+undersized+degraded, 12 stale+active+clean, 24 active+undersized, 108 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 72/627 objects degraded (11.483%) 2024-02-10T06:30:14.351 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:14 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:14.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:14 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:14.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:14 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:14.624 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:14 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:30:14.350+0000 7fccd33e67c0 -1 Falling back to public interface 2024-02-10T06:30:16.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:16 smithi150 ceph-mon[68055]: pgmap v178: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/627 objects degraded (17.384%) 2024-02-10T06:30:16.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:16.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:16.644 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:16 smithi139 ceph-mon[85480]: pgmap v178: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/627 objects degraded (17.384%) 2024-02-10T06:30:16.644 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:16.644 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:16.644 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:16 smithi139 ceph-mon[87870]: pgmap v178: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/627 objects degraded (17.384%) 2024-02-10T06:30:16.644 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:16.644 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:16 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:17.375 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:30:17.060+0000 7fccd33e67c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-02-10T06:30:17.375 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:30:17.093+0000 7fccd33e67c0 -1 osd.4 134 log_to_monitors true 2024-02-10T06:30:18.048 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:18 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:18.048 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:18 smithi150 ceph-mon[68055]: from='osd.4 [v2:172.21.15.150:6800/3964597325,v1:172.21.15.150:6801/3964597325]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:30:18.048 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:18 smithi150 ceph-mon[68055]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:30:18.048 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:18 smithi150 ceph-mon[68055]: pgmap v179: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/627 objects degraded (17.384%) 2024-02-10T06:30:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:18 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:18 smithi139 ceph-mon[85480]: from='osd.4 [v2:172.21.15.150:6800/3964597325,v1:172.21.15.150:6801/3964597325]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:30:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:18 smithi139 ceph-mon[85480]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:30:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:18 smithi139 ceph-mon[85480]: pgmap v179: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/627 objects degraded (17.384%) 2024-02-10T06:30:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:18 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:18 smithi139 ceph-mon[87870]: from='osd.4 [v2:172.21.15.150:6800/3964597325,v1:172.21.15.150:6801/3964597325]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:30:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:18 smithi139 ceph-mon[87870]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:30:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:18 smithi139 ceph-mon[87870]: pgmap v179: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/627 objects degraded (17.384%) 2024-02-10T06:30:18.875 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:30:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:30:18.550+0000 7fccc0836700 -1 osd.4 134 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:30:19.044 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:30:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:30:18] "GET /metrics HTTP/1.1" 200 38202 "" "Prometheus/2.43.0" 2024-02-10T06:30:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[87870]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T06:30:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[87870]: failed to encode map e137 with expected crc 2024-02-10T06:30:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[87870]: osdmap e137: 8 total, 7 up, 8 in 2024-02-10T06:30:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[87870]: failed to encode map e137 with expected crc 2024-02-10T06:30:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[87870]: from='osd.4 [v2:172.21.15.150:6800/3964597325,v1:172.21.15.150:6801/3964597325]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[87870]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[87870]: failed to encode map e137 with expected crc 2024-02-10T06:30:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[87870]: from='osd.4 ' entity='osd.4' 2024-02-10T06:30:19.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[85480]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T06:30:19.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[85480]: failed to encode map e137 with expected crc 2024-02-10T06:30:19.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[85480]: osdmap e137: 8 total, 7 up, 8 in 2024-02-10T06:30:19.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[85480]: failed to encode map e137 with expected crc 2024-02-10T06:30:19.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[85480]: from='osd.4 [v2:172.21.15.150:6800/3964597325,v1:172.21.15.150:6801/3964597325]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:19.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[85480]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:19.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:19.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:19.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[85480]: failed to encode map e137 with expected crc 2024-02-10T06:30:19.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:19 smithi139 ceph-mon[85480]: from='osd.4 ' entity='osd.4' 2024-02-10T06:30:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:19 smithi150 ceph-mon[68055]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T06:30:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:19 smithi150 ceph-mon[68055]: failed to encode map e137 with expected crc 2024-02-10T06:30:19.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:19 smithi150 ceph-mon[68055]: osdmap e137: 8 total, 7 up, 8 in 2024-02-10T06:30:19.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:19 smithi150 ceph-mon[68055]: failed to encode map e137 with expected crc 2024-02-10T06:30:19.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:19 smithi150 ceph-mon[68055]: from='osd.4 [v2:172.21.15.150:6800/3964597325,v1:172.21.15.150:6801/3964597325]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:19.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:19 smithi150 ceph-mon[68055]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:19.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:19 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:19.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:19 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:19.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:19 smithi150 ceph-mon[68055]: failed to encode map e137 with expected crc 2024-02-10T06:30:19.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:19 smithi150 ceph-mon[68055]: from='osd.4 ' entity='osd.4' 2024-02-10T06:30:20.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:20 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:20.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:20 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:20.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:20 smithi150 ceph-mon[68055]: pgmap v181: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 109/627 objects degraded (17.384%) 2024-02-10T06:30:20.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:20 smithi150 ceph-mon[68055]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:30:20.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:20 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:30:20.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:20 smithi150 ceph-mon[68055]: failed to encode map e138 with expected crc 2024-02-10T06:30:20.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:20 smithi150 ceph-mon[68055]: failed to encode map e138 with expected crc 2024-02-10T06:30:20.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:20 smithi150 ceph-mon[68055]: osd.4 [v2:172.21.15.150:6800/3964597325,v1:172.21.15.150:6801/3964597325] boot 2024-02-10T06:30:20.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:20 smithi150 ceph-mon[68055]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T06:30:20.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:20 smithi150 ceph-mon[68055]: failed to encode map e138 with expected crc 2024-02-10T06:30:20.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:20.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:20.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[87870]: pgmap v181: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 109/627 objects degraded (17.384%) 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[87870]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[87870]: failed to encode map e138 with expected crc 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[87870]: failed to encode map e138 with expected crc 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[87870]: osd.4 [v2:172.21.15.150:6800/3964597325,v1:172.21.15.150:6801/3964597325] boot 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[87870]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[87870]: failed to encode map e138 with expected crc 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[85480]: pgmap v181: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 109/627 objects degraded (17.384%) 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[85480]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:30:20.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:30:20.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[85480]: failed to encode map e138 with expected crc 2024-02-10T06:30:20.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[85480]: failed to encode map e138 with expected crc 2024-02-10T06:30:20.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[85480]: osd.4 [v2:172.21.15.150:6800/3964597325,v1:172.21.15.150:6801/3964597325] boot 2024-02-10T06:30:20.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[85480]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T06:30:20.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:20 smithi139 ceph-mon[85480]: failed to encode map e138 with expected crc 2024-02-10T06:30:21.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T06:30:21.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[87870]: pgmap v184: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:21 smithi139 ceph-mon[85480]: pgmap v184: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-02-10T06:30:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T06:30:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: failed to encode map e139 with expected crc 2024-02-10T06:30:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:21 smithi150 ceph-mon[68055]: pgmap v184: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-02-10T06:30:22.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:22 smithi139 ceph-mon[85480]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 17 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:22.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:22 smithi139 ceph-mon[87870]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 17 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:22.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:22 smithi150 ceph-mon[68055]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 17 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:23.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:30:22 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:30:22.953+0000 7ff625973700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-02-10T06:30:24.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-02-10T06:30:24.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[85480]: pgmap v185: 161 pgs: 5 active+undersized+degraded, 11 active+undersized, 145 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 20/627 objects degraded (3.190%) 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-02-10T06:30:24.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:23 smithi139 ceph-mon[87870]: pgmap v185: 161 pgs: 5 active+undersized+degraded, 11 active+undersized, 145 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 20/627 objects degraded (3.190%) 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:24.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-02-10T06:30:24.126 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:23 smithi150 ceph-mon[68055]: pgmap v185: 161 pgs: 5 active+undersized+degraded, 11 active+undersized, 145 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 20/627 objects degraded (3.190%) 2024-02-10T06:30:25.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:25 smithi139 ceph-mon[85480]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:30:25.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:25 smithi139 ceph-mon[85480]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/627 objects degraded (3.190%), 5 pgs degraded) 2024-02-10T06:30:25.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:25 smithi139 ceph-mon[85480]: Cluster is now healthy 2024-02-10T06:30:25.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:25 smithi139 ceph-mon[87870]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:30:25.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:25 smithi139 ceph-mon[87870]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/627 objects degraded (3.190%), 5 pgs degraded) 2024-02-10T06:30:25.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:25 smithi139 ceph-mon[87870]: Cluster is now healthy 2024-02-10T06:30:25.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:25 smithi150 ceph-mon[68055]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:30:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:25 smithi150 ceph-mon[68055]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/627 objects degraded (3.190%), 5 pgs degraded) 2024-02-10T06:30:25.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:25 smithi150 ceph-mon[68055]: Cluster is now healthy 2024-02-10T06:30:28.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:28 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:28.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:28 smithi139 ceph-mon[87870]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:30:28.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:28 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:28.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:28 smithi139 ceph-mon[85480]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:30:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:28 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:28 smithi150 ceph-mon[68055]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:30:29.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:30:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:30:28] "GET /metrics HTTP/1.1" 200 38202 "" "Prometheus/2.43.0" 2024-02-10T06:30:30.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:30 smithi139 ceph-mon[85480]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:30:30.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:30 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:30 smithi139 ceph-mon[87870]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:30:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:30 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:30.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:30 smithi150 ceph-mon[68055]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:30:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:30 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:31.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:31 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:31 smithi150 ceph-mon[68055]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-02-10T06:30:32.078 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:31 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:32.078 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:31 smithi139 ceph-mon[85480]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-02-10T06:30:32.078 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:31 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:32.078 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:31 smithi139 ceph-mon[87870]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-02-10T06:30:32.079 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:30:32.720 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (7m) 49s ago 29m 26.3M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (7m) 14s ago 29m 88.2M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (7m) 49s ago 28m 64.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (8m) 14s ago 34m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (18m) 49s ago 35m 508M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (5m) 49s ago 35m 54.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (6m) 14s ago 34m 48.3M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (5m) 49s ago 34m 41.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (8m) 49s ago 30m 17.0M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (8m) 14s ago 30m 16.7M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (3m) 49s ago 33m 53.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (2m) 49s ago 33m 51.5M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (107s) 49s ago 32m 40.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (53s) 49s ago 32m 12.6M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e1826d3dcabe 2024-02-10T06:30:32.721 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (19s) 14s ago 32m 12.5M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 f61046f26c97 2024-02-10T06:30:32.722 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (31m) 14s ago 31m 60.3M 3248M 17.2.0 e1d6a67b021e 228163c5fc10 2024-02-10T06:30:32.722 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (31m) 14s ago 31m 57.4M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:30:32.722 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (31m) 14s ago 31m 63.2M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:30:32.722 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (7m) 14s ago 30m 65.8M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:30:32.722 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (28m) 49s ago 28m 78.2M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:30:32.722 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (28m) 14s ago 28m 77.7M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:30:33.107 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:30:33.107 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:30:33.107 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:30:33.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:30:33.107 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 10 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:30:33.108 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:30:33.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:33 smithi139 ceph-mon[85480]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:33.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:33 smithi139 ceph-mon[85480]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:33.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:33 smithi139 ceph-mon[87870]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:33.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:33 smithi139 ceph-mon[87870]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:33 smithi150 ceph-mon[68055]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:33.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:33 smithi150 ceph-mon[68055]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:33.438 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:30:33.439 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:30:33.439 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:30:33.439 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-10T06:30:33.439 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:30:33.439 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "5/8 daemons upgraded", 2024-02-10T06:30:33.439 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:30:33.439 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:30:33.439 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:30:34.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:34 smithi139 ceph-mon[85480]: from='client.54522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:34.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:34 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/1457155849' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:34.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:34 smithi139 ceph-mon[85480]: from='client.54534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:34.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:34 smithi139 ceph-mon[85480]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:34 smithi139 ceph-mon[87870]: from='client.54522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:34 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/1457155849' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:34 smithi139 ceph-mon[87870]: from='client.54534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:34.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:34 smithi139 ceph-mon[87870]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:34 smithi150 ceph-mon[68055]: from='client.54522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:34.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:34 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/1457155849' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:34.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:34 smithi150 ceph-mon[68055]: from='client.54534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:30:34.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:34 smithi150 ceph-mon[68055]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:36.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:36 smithi139 ceph-mon[85480]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:36.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:36 smithi139 ceph-mon[87870]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:36.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:36 smithi150 ceph-mon[68055]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:38.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:38 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:38.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:38 smithi139 ceph-mon[87870]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:38.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:38 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:38 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:38 smithi139 ceph-mon[85480]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:38 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:38.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:38 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:38.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:38 smithi150 ceph-mon[68055]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:38 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:39.048 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:30:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:30:38] "GET /metrics HTTP/1.1" 200 38202 "" "Prometheus/2.43.0" 2024-02-10T06:30:39.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:39 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:39.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:39 smithi150 ceph-mon[68055]: Upgrade: osd.5 is safe to restart 2024-02-10T06:30:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:39 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:39.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:39 smithi139 ceph-mon[85480]: Upgrade: osd.5 is safe to restart 2024-02-10T06:30:39.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:39 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:30:39.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:39 smithi139 ceph-mon[87870]: Upgrade: osd.5 is safe to restart 2024-02-10T06:30:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:40 smithi139 ceph-mon[85480]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:40 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:40.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:40 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:30:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:40 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:40 smithi139 ceph-mon[87870]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:40 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:40 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:30:40.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:40 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:40.341 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:40 smithi150 ceph-mon[68055]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:40.341 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:40 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:40.341 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:40 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:30:40.341 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:40 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:41 smithi139 ceph-mon[85480]: Upgrade: Updating osd.5 2024-02-10T06:30:41.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:41 smithi139 ceph-mon[85480]: Deploying daemon osd.5 on smithi150 2024-02-10T06:30:41.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:41 smithi139 ceph-mon[87870]: Upgrade: Updating osd.5 2024-02-10T06:30:41.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:41 smithi139 ceph-mon[87870]: Deploying daemon osd.5 on smithi150 2024-02-10T06:30:41.359 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:41 smithi150 ceph-mon[68055]: Upgrade: Updating osd.5 2024-02-10T06:30:41.359 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:41 smithi150 ceph-mon[68055]: Deploying daemon osd.5 on smithi150 2024-02-10T06:30:42.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:42 smithi139 ceph-mon[85480]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:42.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:42 smithi139 ceph-mon[87870]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:42.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:42 smithi150 ceph-mon[68055]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:42.995 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:42 smithi150 systemd[1]: Stopping Ceph osd.5 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:30:43.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:43 smithi139 ceph-mon[85480]: osd.5 marked itself down and dead 2024-02-10T06:30:43.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:43 smithi139 ceph-mon[87870]: osd.5 marked itself down and dead 2024-02-10T06:30:43.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:43 smithi150 ceph-mon[68055]: osd.5 marked itself down and dead 2024-02-10T06:30:43.375 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:42 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[31899]: 2024-02-10T06:30:42.993+0000 7f164820f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:30:43.375 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:42 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[31899]: 2024-02-10T06:30:42.993+0000 7f164820f700 -1 osd.5 139 *** Got signal Terminated *** 2024-02-10T06:30:43.375 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:42 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[31899]: 2024-02-10T06:30:42.993+0000 7f164820f700 -1 osd.5 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:30:43.803 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:43 smithi150 podman[82902]: 2024-02-10 06:30:43.431683604 +0000 UTC m=+0.539950275 container died 228163c5fc105584ab295a2ac4ef16a217d13e6db08bbd8ce6982f12442993c0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5, version=8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.component=centos-stream-container, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, ceph=True, io.openshift.expose-services=, vcs-type=git, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T06:30:44.067 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:43 smithi150 podman[82902]: 2024-02-10 06:30:43.893876512 +0000 UTC m=+1.002143194 container cleanup 228163c5fc105584ab295a2ac4ef16a217d13e6db08bbd8ce6982f12442993c0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, version=8, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., name=centos-stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T06:30:44.068 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:43 smithi150 bash[82902]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5 2024-02-10T06:30:44.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:30:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: osdmap e140: 8 total, 7 up, 8 in 2024-02-10T06:30:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[85480]: pgmap v196: 161 pgs: 9 peering, 17 stale+active+clean, 135 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T06:30:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:30:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: osdmap e140: 8 total, 7 up, 8 in 2024-02-10T06:30:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:44 smithi139 ceph-mon[87870]: pgmap v196: 161 pgs: 9 peering, 17 stale+active+clean, 135 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T06:30:44.374 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:44 smithi150 podman[82929]: 2024-02-10 06:30:44.212550861 +0000 UTC m=+0.777024989 container remove 228163c5fc105584ab295a2ac4ef16a217d13e6db08bbd8ce6982f12442993c0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, release=754, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: osdmap e140: 8 total, 7 up, 8 in 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: failed to encode map e140 with expected crc 2024-02-10T06:30:44.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:44 smithi150 ceph-mon[68055]: pgmap v196: 161 pgs: 9 peering, 17 stale+active+clean, 135 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T06:30:44.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:44 smithi150 podman[83023]: 2024-02-10 06:30:44.458398251 +0000 UTC m=+0.017468577 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:30:44.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:44 smithi150 podman[83023]: 2024-02-10 06:30:44.580197341 +0000 UTC m=+0.139267654 container create fab2bdc7d476dd44ac671d74755e5cee5abfe2038188bbaf1c800adc79428e34 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-10T06:30:44.859 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:44 smithi150 podman[83023]: 2024-02-10 06:30:44.783039603 +0000 UTC m=+0.342109917 container init fab2bdc7d476dd44ac671d74755e5cee5abfe2038188bbaf1c800adc79428e34 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-02-10T06:30:44.859 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:44 smithi150 podman[83023]: 2024-02-10 06:30:44.786027091 +0000 UTC m=+0.345097412 container start fab2bdc7d476dd44ac671d74755e5cee5abfe2038188bbaf1c800adc79428e34 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-10T06:30:44.859 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:44 smithi150 podman[83023]: 2024-02-10 06:30:44.857874104 +0000 UTC m=+0.416944450 container attach fab2bdc7d476dd44ac671d74755e5cee5abfe2038188bbaf1c800adc79428e34 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, ceph=True) 2024-02-10T06:30:45.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:45 smithi150 ceph-mon[68055]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:30:45.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:45 smithi150 ceph-mon[68055]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:45 smithi150 ceph-mon[68055]: osdmap e141: 8 total, 7 up, 8 in 2024-02-10T06:30:45.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:45 smithi150 ceph-mon[68055]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:45 smithi150 ceph-mon[68055]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:45 smithi150 ceph-mon[68055]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:45 smithi150 ceph-mon[68055]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.125 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:45 smithi150 podman[83023]: 2024-02-10 06:30:45.084141034 +0000 UTC m=+0.643211340 container died fab2bdc7d476dd44ac671d74755e5cee5abfe2038188bbaf1c800adc79428e34 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-02-10T06:30:45.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[85480]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[85480]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[85480]: osdmap e141: 8 total, 7 up, 8 in 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[85480]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[85480]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[85480]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[85480]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[87870]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[87870]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[87870]: osdmap e141: 8 total, 7 up, 8 in 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[87870]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[87870]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[87870]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:45 smithi139 ceph-mon[87870]: failed to encode map e141 with expected crc 2024-02-10T06:30:45.875 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:45 smithi150 podman[83049]: 2024-02-10 06:30:45.728897419 +0000 UTC m=+0.638878609 container remove fab2bdc7d476dd44ac671d74755e5cee5abfe2038188bbaf1c800adc79428e34 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-290212a, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:30:45.875 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:45 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-10T06:30:45.875 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:45 smithi150 systemd[1]: Stopped Ceph osd.5 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:30:45.875 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:45 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.5.service: Consumed 16.032s CPU time. 2024-02-10T06:30:46.278 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:45 smithi150 systemd[1]: Starting Ceph osd.5 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:30:46.279 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 podman[83154]: 2024-02-10 06:30:46.076614295 +0000 UTC m=+0.017261091 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:30:46.279 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 podman[83154]: 2024-02-10 06:30:46.188633035 +0000 UTC m=+0.129279822 container create c3456ba515814f206ca5fe69af5f0f94fcbf87dcd38c25734cb1d95147bfbdcd (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-10T06:30:46.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:46 smithi150 ceph-mon[68055]: pgmap v198: 161 pgs: 28 peering, 10 stale+active+clean, 123 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:30:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:46 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:46.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:46 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:46.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 podman[83154]: 2024-02-10 06:30:46.399723823 +0000 UTC m=+0.340370620 container init c3456ba515814f206ca5fe69af5f0f94fcbf87dcd38c25734cb1d95147bfbdcd (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-10T06:30:46.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 podman[83154]: 2024-02-10 06:30:46.402922053 +0000 UTC m=+0.343568850 container start c3456ba515814f206ca5fe69af5f0f94fcbf87dcd38c25734cb1d95147bfbdcd (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T06:30:46.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 podman[83154]: 2024-02-10 06:30:46.491254606 +0000 UTC m=+0.431901406 container attach c3456ba515814f206ca5fe69af5f0f94fcbf87dcd38c25734cb1d95147bfbdcd (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, RELEASE=main-290212a, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-10T06:30:46.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:46 smithi139 ceph-mon[85480]: pgmap v198: 161 pgs: 28 peering, 10 stale+active+clean, 123 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:30:46.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:46 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:46.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:46 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:46 smithi139 ceph-mon[87870]: pgmap v198: 161 pgs: 28 peering, 10 stale+active+clean, 123 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:30:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:46 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:46 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:47.125 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate[83168]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T06:30:47.125 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate[83168]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T06:30:47.125 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 bash[83154]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T06:30:47.125 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 bash[83154]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T06:30:47.125 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 bash[83154]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T06:30:47.125 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 bash[83154]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T06:30:47.125 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 bash[83154]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-10T06:30:47.126 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 bash[83154]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T06:30:47.126 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 bash[83154]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-10T06:30:47.126 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate[83168]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T06:30:47.126 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate[83168]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T06:30:47.126 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate[83168]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-10T06:30:47.126 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate[83168]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T06:30:47.126 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate[83168]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-10T06:30:47.126 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:46 smithi150 podman[83154]: 2024-02-10 06:30:46.867169722 +0000 UTC m=+0.807816515 container died c3456ba515814f206ca5fe69af5f0f94fcbf87dcd38c25734cb1d95147bfbdcd (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:30:47.752 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:47 smithi150 podman[83284]: 2024-02-10 06:30:47.429106434 +0000 UTC m=+0.555933777 container remove c3456ba515814f206ca5fe69af5f0f94fcbf87dcd38c25734cb1d95147bfbdcd (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-10T06:30:47.752 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:47 smithi150 podman[83351]: 2024-02-10 06:30:47.546548715 +0000 UTC m=+0.017837017 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:30:47.752 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:47 smithi150 podman[83351]: 2024-02-10 06:30:47.655240279 +0000 UTC m=+0.126528566 container create 24270c0f3351c4889b6b29bf02d3ae59dd0352facb3bd1178d7c62e617fe2f2d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-10T06:30:48.050 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:47 smithi150 podman[83351]: 2024-02-10 06:30:47.857414618 +0000 UTC m=+0.328702915 container init 24270c0f3351c4889b6b29bf02d3ae59dd0352facb3bd1178d7c62e617fe2f2d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-10T06:30:48.050 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:47 smithi150 podman[83351]: 2024-02-10 06:30:47.860308851 +0000 UTC m=+0.331597139 container start 24270c0f3351c4889b6b29bf02d3ae59dd0352facb3bd1178d7c62e617fe2f2d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-10T06:30:48.050 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:47 smithi150 ceph-osd[83370]: -- 172.21.15.150:0/2190699122 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5560a06e31e0 con 0x5560a06b5000 2024-02-10T06:30:48.050 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:47 smithi150 bash[83351]: 24270c0f3351c4889b6b29bf02d3ae59dd0352facb3bd1178d7c62e617fe2f2d 2024-02-10T06:30:48.050 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:47 smithi150 systemd[1]: Started Ceph osd.5 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:30:48.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:48 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:48.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:48 smithi139 ceph-mon[85480]: pgmap v199: 161 pgs: 6 active+undersized+degraded, 28 peering, 5 stale+active+clean, 9 active+undersized, 113 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 17/627 objects degraded (2.711%) 2024-02-10T06:30:48.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:48 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:48.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:48 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:48.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:48 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:48 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:48 smithi139 ceph-mon[87870]: pgmap v199: 161 pgs: 6 active+undersized+degraded, 28 peering, 5 stale+active+clean, 9 active+undersized, 113 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 17/627 objects degraded (2.711%) 2024-02-10T06:30:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:48 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:48 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:48 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:48 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:48 smithi150 ceph-mon[68055]: pgmap v199: 161 pgs: 6 active+undersized+degraded, 28 peering, 5 stale+active+clean, 9 active+undersized, 113 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 17/627 objects degraded (2.711%) 2024-02-10T06:30:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:48 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:48 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:48 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:48.875 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:30:48.398+0000 7f3c53d007c0 -1 Falling back to public interface 2024-02-10T06:30:49.051 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:30:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:30:48] "GET /metrics HTTP/1.1" 200 38197 "" "Prometheus/2.43.0" 2024-02-10T06:30:49.194 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:49 smithi150 ceph-mon[68055]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:49.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:49 smithi139 ceph-mon[85480]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:49.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:49 smithi139 ceph-mon[87870]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:50.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:50 smithi150 ceph-mon[68055]: pgmap v200: 161 pgs: 19 active+undersized+degraded, 38 active+undersized, 104 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-02-10T06:30:50.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:50 smithi139 ceph-mon[85480]: pgmap v200: 161 pgs: 19 active+undersized+degraded, 38 active+undersized, 104 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-02-10T06:30:50.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:50 smithi139 ceph-mon[87870]: pgmap v200: 161 pgs: 19 active+undersized+degraded, 38 active+undersized, 104 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-02-10T06:30:51.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:51 smithi139 ceph-mon[85480]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 7 pgs peering) 2024-02-10T06:30:51.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:51 smithi139 ceph-mon[87870]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 7 pgs peering) 2024-02-10T06:30:51.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:51 smithi150 ceph-mon[68055]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 7 pgs peering) 2024-02-10T06:30:51.375 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:30:51.109+0000 7f3c53d007c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-02-10T06:30:51.375 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:30:51.133+0000 7f3c53d007c0 -1 osd.5 139 log_to_monitors true 2024-02-10T06:30:52.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:52 smithi139 ceph-mon[85480]: from='osd.5 [v2:172.21.15.150:6808/762642710,v1:172.21.15.150:6809/762642710]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:30:52.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:52 smithi139 ceph-mon[85480]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:30:52.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:52 smithi139 ceph-mon[85480]: pgmap v201: 161 pgs: 19 active+undersized+degraded, 38 active+undersized, 104 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-02-10T06:30:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:52 smithi139 ceph-mon[87870]: from='osd.5 [v2:172.21.15.150:6808/762642710,v1:172.21.15.150:6809/762642710]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:30:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:52 smithi139 ceph-mon[87870]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:30:52.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:52 smithi139 ceph-mon[87870]: pgmap v201: 161 pgs: 19 active+undersized+degraded, 38 active+undersized, 104 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-02-10T06:30:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:52 smithi150 ceph-mon[68055]: from='osd.5 [v2:172.21.15.150:6808/762642710,v1:172.21.15.150:6809/762642710]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:30:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:52 smithi150 ceph-mon[68055]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:30:52.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:52 smithi150 ceph-mon[68055]: pgmap v201: 161 pgs: 19 active+undersized+degraded, 38 active+undersized, 104 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-02-10T06:30:52.375 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:30:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:30:52.097+0000 7f3c45158700 -1 osd.5 139 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:30:53.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[85480]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T06:30:53.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[85480]: failed to encode map e142 with expected crc 2024-02-10T06:30:53.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[85480]: osdmap e142: 8 total, 7 up, 8 in 2024-02-10T06:30:53.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[85480]: from='osd.5 [v2:172.21.15.150:6808/762642710,v1:172.21.15.150:6809/762642710]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:53.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[85480]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:53.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[85480]: failed to encode map e142 with expected crc 2024-02-10T06:30:53.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:53.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[87870]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T06:30:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[87870]: failed to encode map e142 with expected crc 2024-02-10T06:30:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[87870]: osdmap e142: 8 total, 7 up, 8 in 2024-02-10T06:30:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[87870]: from='osd.5 [v2:172.21.15.150:6808/762642710,v1:172.21.15.150:6809/762642710]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[87870]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[87870]: failed to encode map e142 with expected crc 2024-02-10T06:30:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:53 smithi150 ceph-mon[68055]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T06:30:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:53 smithi150 ceph-mon[68055]: failed to encode map e142 with expected crc 2024-02-10T06:30:53.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:53 smithi150 ceph-mon[68055]: osdmap e142: 8 total, 7 up, 8 in 2024-02-10T06:30:53.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:53 smithi150 ceph-mon[68055]: from='osd.5 [v2:172.21.15.150:6808/762642710,v1:172.21.15.150:6809/762642710]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:53.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:53 smithi150 ceph-mon[68055]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:30:53.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:53 smithi150 ceph-mon[68055]: failed to encode map e142 with expected crc 2024-02-10T06:30:53.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:53.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:54.335 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:54 smithi150 ceph-mon[68055]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:30:54.335 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:54 smithi150 ceph-mon[68055]: failed to encode map e143 with expected crc 2024-02-10T06:30:54.335 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:54 smithi150 ceph-mon[68055]: failed to encode map e143 with expected crc 2024-02-10T06:30:54.335 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:54 smithi150 ceph-mon[68055]: osd.5 [v2:172.21.15.150:6808/762642710,v1:172.21.15.150:6809/762642710] boot 2024-02-10T06:30:54.335 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:54 smithi150 ceph-mon[68055]: osdmap e143: 8 total, 8 up, 8 in 2024-02-10T06:30:54.335 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:54 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:30:54.335 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:54 smithi150 ceph-mon[68055]: pgmap v204: 161 pgs: 5 peering, 18 active+undersized+degraded, 34 active+undersized, 104 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 70/627 objects degraded (11.164%) 2024-02-10T06:30:54.336 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:54 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:54.336 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:54 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[87870]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:30:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[87870]: failed to encode map e143 with expected crc 2024-02-10T06:30:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[87870]: failed to encode map e143 with expected crc 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[87870]: osd.5 [v2:172.21.15.150:6808/762642710,v1:172.21.15.150:6809/762642710] boot 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[87870]: osdmap e143: 8 total, 8 up, 8 in 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[87870]: pgmap v204: 161 pgs: 5 peering, 18 active+undersized+degraded, 34 active+undersized, 104 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 70/627 objects degraded (11.164%) 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[85480]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[85480]: failed to encode map e143 with expected crc 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[85480]: failed to encode map e143 with expected crc 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[85480]: osd.5 [v2:172.21.15.150:6808/762642710,v1:172.21.15.150:6809/762642710] boot 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[85480]: osdmap e143: 8 total, 8 up, 8 in 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:30:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[85480]: pgmap v204: 161 pgs: 5 peering, 18 active+undersized+degraded, 34 active+undersized, 104 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 70/627 objects degraded (11.164%) 2024-02-10T06:30:54.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:54.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:54 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:55.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[85480]: Health check update: Degraded data redundancy: 70/627 objects degraded (11.164%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:55.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[85480]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[85480]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[85480]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[85480]: osdmap e144: 8 total, 8 up, 8 in 2024-02-10T06:30:55.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[85480]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[85480]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[85480]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[85480]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[87870]: Health check update: Degraded data redundancy: 70/627 objects degraded (11.164%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:55.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[87870]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[87870]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[87870]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[87870]: osdmap e144: 8 total, 8 up, 8 in 2024-02-10T06:30:55.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[87870]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[87870]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[87870]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:55 smithi139 ceph-mon[87870]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:55 smithi150 ceph-mon[68055]: Health check update: Degraded data redundancy: 70/627 objects degraded (11.164%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T06:30:55.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:55 smithi150 ceph-mon[68055]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:55 smithi150 ceph-mon[68055]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:55 smithi150 ceph-mon[68055]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:55 smithi150 ceph-mon[68055]: osdmap e144: 8 total, 8 up, 8 in 2024-02-10T06:30:55.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:55 smithi150 ceph-mon[68055]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:55 smithi150 ceph-mon[68055]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:55 smithi150 ceph-mon[68055]: failed to encode map e144 with expected crc 2024-02-10T06:30:55.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:55 smithi150 ceph-mon[68055]: failed to encode map e144 with expected crc 2024-02-10T06:30:56.344 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:56 smithi150 ceph-mon[68055]: pgmap v206: 161 pgs: 17 peering, 12 active+undersized+degraded, 28 active+undersized, 104 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-02-10T06:30:56.588 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:56 smithi139 ceph-mon[85480]: pgmap v206: 161 pgs: 17 peering, 12 active+undersized+degraded, 28 active+undersized, 104 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-02-10T06:30:56.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:56 smithi139 ceph-mon[87870]: pgmap v206: 161 pgs: 17 peering, 12 active+undersized+degraded, 28 active+undersized, 104 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-02-10T06:30:57.586 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:30:57 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:30:57.200+0000 7ff625973700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:30:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[85480]: pgmap v207: 161 pgs: 17 peering, 1 active+undersized+degraded, 8 active+undersized, 135 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 10/627 objects degraded (1.595%) 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:30:58.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:30:58.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-10T06:30:58.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-mon[87870]: pgmap v207: 161 pgs: 17 peering, 1 active+undersized+degraded, 8 active+undersized, 135 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 10/627 objects degraded (1.595%) 2024-02-10T06:30:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-10T06:30:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:58 smithi150 ceph-mon[68055]: pgmap v207: 161 pgs: 17 peering, 1 active+undersized+degraded, 8 active+undersized, 135 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 10/627 objects degraded (1.595%) 2024-02-10T06:30:59.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:30:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:30:58] "GET /metrics HTTP/1.1" 200 38197 "" "Prometheus/2.43.0" 2024-02-10T06:30:59.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:59 smithi139 ceph-mon[85480]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:30:59.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:59 smithi139 ceph-mon[85480]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/627 objects degraded (1.595%), 1 pg degraded) 2024-02-10T06:30:59.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:30:59 smithi139 ceph-mon[85480]: Cluster is now healthy 2024-02-10T06:30:59.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:59 smithi139 ceph-mon[87870]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:30:59.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:59 smithi139 ceph-mon[87870]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/627 objects degraded (1.595%), 1 pg degraded) 2024-02-10T06:30:59.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:30:59 smithi139 ceph-mon[87870]: Cluster is now healthy 2024-02-10T06:30:59.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:59 smithi150 ceph-mon[68055]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:30:59.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:59 smithi150 ceph-mon[68055]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/627 objects degraded (1.595%), 1 pg degraded) 2024-02-10T06:30:59.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:30:59 smithi150 ceph-mon[68055]: Cluster is now healthy 2024-02-10T06:31:01.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:01.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:01.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:01.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:02.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:02 smithi139 ceph-mon[85480]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-02-10T06:31:02.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:02 smithi139 ceph-mon[87870]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-02-10T06:31:02.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:02 smithi150 ceph-mon[68055]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-02-10T06:31:03.796 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:31:03.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:03 smithi139 ceph-mon[85480]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-02-10T06:31:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:03 smithi139 ceph-mon[87870]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-02-10T06:31:03.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:03 smithi150 ceph-mon[68055]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-02-10T06:31:04.432 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (8m) 80s ago 30m 26.3M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (7m) 12s ago 29m 90.1M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (8m) 80s ago 29m 64.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (9m) 12s ago 34m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (19m) 80s ago 36m 508M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (5m) 80s ago 36m 54.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (6m) 12s ago 34m 49.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (5m) 80s ago 35m 41.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (8m) 80s ago 30m 17.0M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (8m) 12s ago 30m 16.4M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (3m) 80s ago 34m 53.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (3m) 80s ago 33m 51.5M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (2m) 80s ago 33m 40.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (85s) 80s ago 33m 12.6M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e1826d3dcabe 2024-02-10T06:31:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (50s) 12s ago 32m 52.0M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 f61046f26c97 2024-02-10T06:31:04.434 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (16s) 12s ago 32m 11.8M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 24270c0f3351 2024-02-10T06:31:04.434 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (32m) 12s ago 32m 59.6M 3248M 17.2.0 e1d6a67b021e 7c72ec30be2b 2024-02-10T06:31:04.434 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (31m) 12s ago 31m 64.2M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:31:04.434 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (8m) 12s ago 30m 65.8M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:31:04.434 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (29m) 80s ago 29m 78.2M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:31:04.434 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (29m) 12s ago 29m 77.9M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:31:04.738 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:04 smithi139 ceph-mon[85480]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:04.738 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:04 smithi139 ceph-mon[85480]: from='client.54549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:04.738 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:04 smithi139 ceph-mon[85480]: from='client.64489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:04.738 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:04 smithi139 ceph-mon[87870]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:04.738 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:04 smithi139 ceph-mon[87870]: from='client.54549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:04.738 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:04 smithi139 ceph-mon[87870]: from='client.64489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:04.824 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:31:04.824 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:31:04.824 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:31:04.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:31:04.824 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 6 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 11 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:31:04.825 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:31:04.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:04 smithi150 ceph-mon[68055]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:04.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:04 smithi150 ceph-mon[68055]: from='client.54549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:04.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:04 smithi150 ceph-mon[68055]: from='client.64489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:05.150 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:31:05.150 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:31:05.150 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:31:05.150 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-10T06:31:05.150 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:31:05.150 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "6/8 daemons upgraded", 2024-02-10T06:31:05.150 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:31:05.150 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:31:05.150 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:31:05.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:05 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/2805324821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:05.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:05 smithi139 ceph-mon[85480]: from='client.54564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:05.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:05 smithi139 ceph-mon[85480]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-02-10T06:31:05.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:05 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/2805324821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:05.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:05 smithi139 ceph-mon[87870]: from='client.54564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:05.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:05 smithi139 ceph-mon[87870]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-02-10T06:31:05.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:05 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/2805324821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:05.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:05 smithi150 ceph-mon[68055]: from='client.54564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:05.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:05 smithi150 ceph-mon[68055]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-02-10T06:31:06.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:06 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:07.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:06 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:07.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:06 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:08.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:08 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:08.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:08 smithi139 ceph-mon[85480]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:08 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:08 smithi139 ceph-mon[87870]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:08.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:08 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:08 smithi150 ceph-mon[68055]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:09.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:31:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:31:08] "GET /metrics HTTP/1.1" 200 38212 "" "Prometheus/2.43.0" 2024-02-10T06:31:09.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:09 smithi139 ceph-mon[85480]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:09.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:09 smithi139 ceph-mon[87870]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:09.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:09 smithi150 ceph-mon[68055]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:11.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:11 smithi139 ceph-mon[85480]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:11.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:11 smithi139 ceph-mon[87870]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:11.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:11 smithi150 ceph-mon[68055]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:13.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:13 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:31:13.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:13 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:31:13.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:13 smithi139 ceph-mon[85480]: Upgrade: osd.6 is safe to restart 2024-02-10T06:31:13.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:13 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:31:13.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:13 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:31:13.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:13 smithi139 ceph-mon[87870]: Upgrade: osd.6 is safe to restart 2024-02-10T06:31:13.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:13 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:31:13.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:13 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:31:13.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:13 smithi150 ceph-mon[68055]: Upgrade: osd.6 is safe to restart 2024-02-10T06:31:14.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:14 smithi139 ceph-mon[85480]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:14.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:14 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:14.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:14 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:31:14.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:14 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:14.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:14 smithi139 ceph-mon[87870]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:14.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:14 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:14.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:14 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:31:14.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:14 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:14.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:14 smithi150 ceph-mon[68055]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:14.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:14 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:14.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:14 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:31:14.353 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:14 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:15.264 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:15 smithi150 ceph-mon[68055]: Upgrade: Updating osd.6 2024-02-10T06:31:15.264 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:15 smithi150 ceph-mon[68055]: Deploying daemon osd.6 on smithi150 2024-02-10T06:31:15.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:15 smithi139 ceph-mon[85480]: Upgrade: Updating osd.6 2024-02-10T06:31:15.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:15 smithi139 ceph-mon[85480]: Deploying daemon osd.6 on smithi150 2024-02-10T06:31:15.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:15 smithi139 ceph-mon[87870]: Upgrade: Updating osd.6 2024-02-10T06:31:15.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:15 smithi139 ceph-mon[87870]: Deploying daemon osd.6 on smithi150 2024-02-10T06:31:16.255 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:16.255 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:16 smithi150 ceph-mon[68055]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:16.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:16 smithi139 ceph-mon[85480]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:16.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:16 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:16.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:16 smithi139 ceph-mon[87870]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:17.169 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:16 smithi150 systemd[1]: Stopping Ceph osd.6 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:31:17.624 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[34777]: 2024-02-10T06:31:17.168+0000 7fb224074700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:31:17.625 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[34777]: 2024-02-10T06:31:17.168+0000 7fb224074700 -1 osd.6 144 *** Got signal Terminated *** 2024-02-10T06:31:17.625 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:17 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[34777]: 2024-02-10T06:31:17.168+0000 7fb224074700 -1 osd.6 144 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:31:18.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:18 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:18 smithi139 ceph-mon[85480]: osd.6 marked itself down and dead 2024-02-10T06:31:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:18 smithi139 ceph-mon[85480]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:31:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:18 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:18 smithi139 ceph-mon[87870]: osd.6 marked itself down and dead 2024-02-10T06:31:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:18 smithi139 ceph-mon[87870]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:31:18.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:18 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:18.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:18 smithi150 ceph-mon[68055]: osd.6 marked itself down and dead 2024-02-10T06:31:18.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:18 smithi150 ceph-mon[68055]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:31:18.811 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:18 smithi150 podman[87860]: 2024-02-10 06:31:18.512544546 +0000 UTC m=+1.463303153 container died 7c72ec30be2bfa14120088a7b0d228ba854af519f2ee1fd160b955c4484a895a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, ceph=True, io.openshift.expose-services=, release=754, maintainer=Guillaume Abrioux , name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vcs-type=git, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-10T06:31:19.058 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:31:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:31:18] "GET /metrics HTTP/1.1" 200 38214 "" "Prometheus/2.43.0" 2024-02-10T06:31:19.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:19 smithi150 ceph-mon[68055]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:31:19.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:19 smithi150 ceph-mon[68055]: osdmap e145: 8 total, 7 up, 8 in 2024-02-10T06:31:19.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:19 smithi150 ceph-mon[68055]: failed to encode map e145 with expected crc 2024-02-10T06:31:19.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:18 smithi150 podman[87860]: 2024-02-10 06:31:18.902069015 +0000 UTC m=+1.852827622 container cleanup 7c72ec30be2bfa14120088a7b0d228ba854af519f2ee1fd160b955c4484a895a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.expose-services=, distribution-scope=public, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vcs-type=git, release=754, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-10T06:31:19.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:18 smithi150 bash[87860]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6 2024-02-10T06:31:19.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:19 smithi139 ceph-mon[85480]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:31:19.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:19 smithi139 ceph-mon[85480]: osdmap e145: 8 total, 7 up, 8 in 2024-02-10T06:31:19.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:19 smithi139 ceph-mon[85480]: failed to encode map e145 with expected crc 2024-02-10T06:31:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:19 smithi139 ceph-mon[87870]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:31:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:19 smithi139 ceph-mon[87870]: osdmap e145: 8 total, 7 up, 8 in 2024-02-10T06:31:19.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:19 smithi139 ceph-mon[87870]: failed to encode map e145 with expected crc 2024-02-10T06:31:19.469 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:19 smithi150 podman[87889]: 2024-02-10 06:31:19.153264208 +0000 UTC m=+0.641094041 container remove 7c72ec30be2bfa14120088a7b0d228ba854af519f2ee1fd160b955c4484a895a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6, GIT_BRANCH=HEAD, ceph=True, com.redhat.component=centos-stream-container, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, distribution-scope=public, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, architecture=x86_64, name=centos-stream, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8) 2024-02-10T06:31:19.757 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:19 smithi150 podman[87984]: 2024-02-10 06:31:19.371992122 +0000 UTC m=+0.018107511 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:31:19.757 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:19 smithi150 podman[87984]: 2024-02-10 06:31:19.479185035 +0000 UTC m=+0.125300485 container create 7467d6dcba84a4d3d1c4fbcd9e9a044a9f4a85f65f1cf8ac4cb4a1d74eafd3e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-02-10T06:31:19.758 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:19 smithi150 podman[87984]: 2024-02-10 06:31:19.68194514 +0000 UTC m=+0.328060528 container init 7467d6dcba84a4d3d1c4fbcd9e9a044a9f4a85f65f1cf8ac4cb4a1d74eafd3e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, RELEASE=main-290212a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-02-10T06:31:19.758 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:19 smithi150 podman[87984]: 2024-02-10 06:31:19.685048264 +0000 UTC m=+0.331163659 container start 7467d6dcba84a4d3d1c4fbcd9e9a044a9f4a85f65f1cf8ac4cb4a1d74eafd3e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-10T06:31:20.070 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:19 smithi150 podman[87984]: 2024-02-10 06:31:19.756940763 +0000 UTC m=+0.403056170 container attach 7467d6dcba84a4d3d1c4fbcd9e9a044a9f4a85f65f1cf8ac4cb4a1d74eafd3e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-10T06:31:20.070 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:19 smithi150 podman[87984]: 2024-02-10 06:31:19.974453219 +0000 UTC m=+0.620568609 container died 7467d6dcba84a4d3d1c4fbcd9e9a044a9f4a85f65f1cf8ac4cb4a1d74eafd3e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-10T06:31:20.327 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:20 smithi150 ceph-mon[68055]: failed to encode map e146 with expected crc 2024-02-10T06:31:20.327 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:20 smithi150 ceph-mon[68055]: osdmap e146: 8 total, 7 up, 8 in 2024-02-10T06:31:20.327 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:20 smithi150 ceph-mon[68055]: pgmap v220: 161 pgs: 30 peering, 4 stale+active+clean, 127 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:31:20.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:20 smithi139 ceph-mon[85480]: failed to encode map e146 with expected crc 2024-02-10T06:31:20.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:20 smithi139 ceph-mon[85480]: osdmap e146: 8 total, 7 up, 8 in 2024-02-10T06:31:20.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:20 smithi139 ceph-mon[85480]: pgmap v220: 161 pgs: 30 peering, 4 stale+active+clean, 127 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:31:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:20 smithi139 ceph-mon[87870]: failed to encode map e146 with expected crc 2024-02-10T06:31:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:20 smithi139 ceph-mon[87870]: osdmap e146: 8 total, 7 up, 8 in 2024-02-10T06:31:20.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:20 smithi139 ceph-mon[87870]: pgmap v220: 161 pgs: 30 peering, 4 stale+active+clean, 127 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:31:20.875 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:20 smithi150 podman[88010]: 2024-02-10 06:31:20.594721894 +0000 UTC m=+0.614826399 container remove 7467d6dcba84a4d3d1c4fbcd9e9a044a9f4a85f65f1cf8ac4cb4a1d74eafd3e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-10T06:31:20.875 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:20 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.6.service: Deactivated successfully. 2024-02-10T06:31:20.875 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:20 smithi150 systemd[1]: Stopped Ceph osd.6 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:31:20.875 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:20 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.6.service: Consumed 14.829s CPU time. 2024-02-10T06:31:20.876 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:20 smithi150 systemd[1]: Starting Ceph osd.6 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:31:21.298 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:21 smithi150 ceph-mon[68055]: Health check failed: Reduced data availability: 7 pgs peering (PG_AVAILABILITY) 2024-02-10T06:31:21.299 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 podman[88116]: 2024-02-10 06:31:20.94890375 +0000 UTC m=+0.018459139 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:31:21.299 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 podman[88116]: 2024-02-10 06:31:21.070907581 +0000 UTC m=+0.140462961 container create 593b229a20eff6bc7840a002725e7acf44f115e762ac2319f460f49784a9bc56 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-10T06:31:21.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:21 smithi139 ceph-mon[85480]: Health check failed: Reduced data availability: 7 pgs peering (PG_AVAILABILITY) 2024-02-10T06:31:21.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:21 smithi139 ceph-mon[87870]: Health check failed: Reduced data availability: 7 pgs peering (PG_AVAILABILITY) 2024-02-10T06:31:21.625 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 podman[88116]: 2024-02-10 06:31:21.298507763 +0000 UTC m=+0.368063142 container init 593b229a20eff6bc7840a002725e7acf44f115e762ac2319f460f49784a9bc56 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-10T06:31:21.625 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 podman[88116]: 2024-02-10 06:31:21.301837405 +0000 UTC m=+0.371392788 container start 593b229a20eff6bc7840a002725e7acf44f115e762ac2319f460f49784a9bc56 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:31:21.625 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 podman[88116]: 2024-02-10 06:31:21.37362754 +0000 UTC m=+0.443182932 container attach 593b229a20eff6bc7840a002725e7acf44f115e762ac2319f460f49784a9bc56 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2) 2024-02-10T06:31:22.077 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate[88130]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T06:31:22.077 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate[88130]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T06:31:22.077 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 bash[88116]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T06:31:22.077 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 bash[88116]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T06:31:22.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 bash[88116]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T06:31:22.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 bash[88116]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T06:31:22.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 bash[88116]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-10T06:31:22.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 bash[88116]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T06:31:22.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 bash[88116]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-10T06:31:22.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate[88130]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T06:31:22.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate[88130]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T06:31:22.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate[88130]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-10T06:31:22.079 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate[88130]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T06:31:22.079 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate[88130]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-10T06:31:22.079 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:21 smithi150 podman[88116]: 2024-02-10 06:31:21.769634426 +0000 UTC m=+0.839189806 container died 593b229a20eff6bc7840a002725e7acf44f115e762ac2319f460f49784a9bc56 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:31:22.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:22 smithi139 ceph-mon[85480]: pgmap v221: 161 pgs: 3 active+undersized+degraded, 30 peering, 3 stale+active+clean, 6 active+undersized, 119 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 15/627 objects degraded (2.392%) 2024-02-10T06:31:22.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:22 smithi139 ceph-mon[87870]: pgmap v221: 161 pgs: 3 active+undersized+degraded, 30 peering, 3 stale+active+clean, 6 active+undersized, 119 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 15/627 objects degraded (2.392%) 2024-02-10T06:31:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:22 smithi150 ceph-mon[68055]: pgmap v221: 161 pgs: 3 active+undersized+degraded, 30 peering, 3 stale+active+clean, 6 active+undersized, 119 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 15/627 objects degraded (2.392%) 2024-02-10T06:31:22.694 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:22 smithi150 podman[88247]: 2024-02-10 06:31:22.394533503 +0000 UTC m=+0.616485887 container remove 593b229a20eff6bc7840a002725e7acf44f115e762ac2319f460f49784a9bc56 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-10T06:31:22.694 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:22 smithi150 podman[88312]: 2024-02-10 06:31:22.595539609 +0000 UTC m=+0.098793550 container create 6136d2cdfcc2355b78394c3fd79c7e5f4ac9b235cd60c00f6b163aab06e67b62 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-10T06:31:22.694 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:22 smithi150 podman[88312]: 2024-02-10 06:31:22.514613699 +0000 UTC m=+0.017867647 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:31:23.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:22 smithi150 podman[88312]: 2024-02-10 06:31:22.789384179 +0000 UTC m=+0.292638122 container init 6136d2cdfcc2355b78394c3fd79c7e5f4ac9b235cd60c00f6b163aab06e67b62 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-02-10T06:31:23.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:22 smithi150 podman[88312]: 2024-02-10 06:31:22.79230446 +0000 UTC m=+0.295558405 container start 6136d2cdfcc2355b78394c3fd79c7e5f4ac9b235cd60c00f6b163aab06e67b62 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2) 2024-02-10T06:31:23.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:22 smithi150 ceph-osd[88331]: -- 172.21.15.150:0/3987231458 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f46077b1e0 con 0x55f46074cc00 2024-02-10T06:31:23.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:22 smithi150 bash[88312]: 6136d2cdfcc2355b78394c3fd79c7e5f4ac9b235cd60c00f6b163aab06e67b62 2024-02-10T06:31:23.078 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:22 smithi150 systemd[1]: Started Ceph osd.6 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:31:23.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:23 smithi150 ceph-mon[68055]: Health check failed: Degraded data redundancy: 15/627 objects degraded (2.392%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T06:31:23.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:23 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:23.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:23 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:23.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:23 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:23.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:23 smithi139 ceph-mon[85480]: Health check failed: Degraded data redundancy: 15/627 objects degraded (2.392%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T06:31:23.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:23 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:23.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:23 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:23.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:23 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:23 smithi139 ceph-mon[87870]: Health check failed: Degraded data redundancy: 15/627 objects degraded (2.392%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T06:31:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:23 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:23 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:23.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:23 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:23.624 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:23 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:31:23.332+0000 7fd614d027c0 -1 Falling back to public interface 2024-02-10T06:31:24.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:24 smithi139 ceph-mon[85480]: pgmap v222: 161 pgs: 16 active+undersized+degraded, 15 peering, 23 active+undersized, 107 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-02-10T06:31:24.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:24 smithi139 ceph-mon[87870]: pgmap v222: 161 pgs: 16 active+undersized+degraded, 15 peering, 23 active+undersized, 107 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-02-10T06:31:24.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:24 smithi150 ceph-mon[68055]: pgmap v222: 161 pgs: 16 active+undersized+degraded, 15 peering, 23 active+undersized, 107 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-02-10T06:31:25.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:25 smithi150 ceph-mon[68055]: pgmap v223: 161 pgs: 20 active+undersized+degraded, 34 active+undersized, 107 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:31:25.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:25 smithi150 ceph-mon[68055]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T06:31:25.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:25 smithi139 ceph-mon[87870]: pgmap v223: 161 pgs: 20 active+undersized+degraded, 34 active+undersized, 107 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:31:25.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:25 smithi139 ceph-mon[87870]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T06:31:25.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:25 smithi139 ceph-mon[85480]: pgmap v223: 161 pgs: 20 active+undersized+degraded, 34 active+undersized, 107 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:31:25.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:25 smithi139 ceph-mon[85480]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T06:31:26.375 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:31:26.069+0000 7fd614d027c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-02-10T06:31:26.375 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:31:26.101+0000 7fd614d027c0 -1 osd.6 144 log_to_monitors true 2024-02-10T06:31:26.836 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:26 smithi150 ceph-mon[68055]: from='osd.6 [v2:172.21.15.150:6816/3147869175,v1:172.21.15.150:6817/3147869175]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:31:26.836 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:26 smithi150 ceph-mon[68055]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:31:26.836 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:31:26 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:31:26.617+0000 7fd60615a700 -1 osd.6 144 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:31:26.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:26 smithi139 ceph-mon[87870]: from='osd.6 [v2:172.21.15.150:6816/3147869175,v1:172.21.15.150:6817/3147869175]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:31:26.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:26 smithi139 ceph-mon[87870]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:31:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:26 smithi139 ceph-mon[85480]: from='osd.6 [v2:172.21.15.150:6816/3147869175,v1:172.21.15.150:6817/3147869175]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:31:26.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:26 smithi139 ceph-mon[85480]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:31:27.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:27 smithi150 ceph-mon[68055]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T06:31:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:27 smithi150 ceph-mon[68055]: osdmap e147: 8 total, 7 up, 8 in 2024-02-10T06:31:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:27 smithi150 ceph-mon[68055]: from='osd.6 [v2:172.21.15.150:6816/3147869175,v1:172.21.15.150:6817/3147869175]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:31:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:27 smithi150 ceph-mon[68055]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:31:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:27 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:27 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:27 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:27.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:27 smithi150 ceph-mon[68055]: pgmap v225: 161 pgs: 20 active+undersized+degraded, 34 active+undersized, 107 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:31:28.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[85480]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T06:31:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[85480]: osdmap e147: 8 total, 7 up, 8 in 2024-02-10T06:31:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[85480]: from='osd.6 [v2:172.21.15.150:6816/3147869175,v1:172.21.15.150:6817/3147869175]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:31:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[85480]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:31:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:28.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[85480]: pgmap v225: 161 pgs: 20 active+undersized+degraded, 34 active+undersized, 107 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:31:28.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[87870]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T06:31:28.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[87870]: osdmap e147: 8 total, 7 up, 8 in 2024-02-10T06:31:28.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[87870]: from='osd.6 [v2:172.21.15.150:6816/3147869175,v1:172.21.15.150:6817/3147869175]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:31:28.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[87870]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:31:28.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:28.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:28.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:28.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:27 smithi139 ceph-mon[87870]: pgmap v225: 161 pgs: 20 active+undersized+degraded, 34 active+undersized, 107 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-02-10T06:31:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: failed to encode map e147 with expected crc 2024-02-10T06:31:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:31:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:31:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: failed to encode map e148 with expected crc 2024-02-10T06:31:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: osd.6 [v2:172.21.15.150:6816/3147869175,v1:172.21.15.150:6817/3147869175] boot 2024-02-10T06:31:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: osdmap e148: 8 total, 8 up, 8 in 2024-02-10T06:31:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: failed to encode map e148 with expected crc 2024-02-10T06:31:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: failed to encode map e148 with expected crc 2024-02-10T06:31:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: failed to encode map e148 with expected crc 2024-02-10T06:31:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: failed to encode map e148 with expected crc 2024-02-10T06:31:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:28.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:28.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:28 smithi150 ceph-mon[68055]: osdmap e149: 8 total, 8 up, 8 in 2024-02-10T06:31:29.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:31:28] "GET /metrics HTTP/1.1" 200 38214 "" "Prometheus/2.43.0" 2024-02-10T06:31:29.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: failed to encode map e147 with expected crc 2024-02-10T06:31:29.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:31:29.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:31:29.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: failed to encode map e148 with expected crc 2024-02-10T06:31:29.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: osd.6 [v2:172.21.15.150:6816/3147869175,v1:172.21.15.150:6817/3147869175] boot 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: osdmap e148: 8 total, 8 up, 8 in 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: failed to encode map e148 with expected crc 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: failed to encode map e148 with expected crc 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: failed to encode map e148 with expected crc 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: failed to encode map e148 with expected crc 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[85480]: osdmap e149: 8 total, 8 up, 8 in 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: failed to encode map e147 with expected crc 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:31:29.088 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: failed to encode map e148 with expected crc 2024-02-10T06:31:29.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: osd.6 [v2:172.21.15.150:6816/3147869175,v1:172.21.15.150:6817/3147869175] boot 2024-02-10T06:31:29.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: osdmap e148: 8 total, 8 up, 8 in 2024-02-10T06:31:29.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: failed to encode map e148 with expected crc 2024-02-10T06:31:29.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: failed to encode map e148 with expected crc 2024-02-10T06:31:29.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: failed to encode map e148 with expected crc 2024-02-10T06:31:29.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: failed to encode map e148 with expected crc 2024-02-10T06:31:29.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:29.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:29.089 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:28 smithi139 ceph-mon[87870]: osdmap e149: 8 total, 8 up, 8 in 2024-02-10T06:31:29.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:29 smithi150 ceph-mon[68055]: failed to encode map e149 with expected crc 2024-02-10T06:31:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:29 smithi150 ceph-mon[68055]: failed to encode map e149 with expected crc 2024-02-10T06:31:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:29 smithi150 ceph-mon[68055]: failed to encode map e149 with expected crc 2024-02-10T06:31:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:29 smithi150 ceph-mon[68055]: failed to encode map e149 with expected crc 2024-02-10T06:31:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:29 smithi150 ceph-mon[68055]: pgmap v228: 161 pgs: 4 activating, 22 peering, 12 active+undersized+degraded, 16 active+undersized, 107 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 43/627 objects degraded (6.858%) 2024-02-10T06:31:30.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:29 smithi139 ceph-mon[85480]: failed to encode map e149 with expected crc 2024-02-10T06:31:30.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:29 smithi139 ceph-mon[85480]: failed to encode map e149 with expected crc 2024-02-10T06:31:30.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:29 smithi139 ceph-mon[85480]: failed to encode map e149 with expected crc 2024-02-10T06:31:30.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:29 smithi139 ceph-mon[85480]: failed to encode map e149 with expected crc 2024-02-10T06:31:30.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:29 smithi139 ceph-mon[85480]: pgmap v228: 161 pgs: 4 activating, 22 peering, 12 active+undersized+degraded, 16 active+undersized, 107 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 43/627 objects degraded (6.858%) 2024-02-10T06:31:30.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:29 smithi139 ceph-mon[87870]: failed to encode map e149 with expected crc 2024-02-10T06:31:30.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:29 smithi139 ceph-mon[87870]: failed to encode map e149 with expected crc 2024-02-10T06:31:30.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:29 smithi139 ceph-mon[87870]: failed to encode map e149 with expected crc 2024-02-10T06:31:30.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:29 smithi139 ceph-mon[87870]: failed to encode map e149 with expected crc 2024-02-10T06:31:30.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:29 smithi139 ceph-mon[87870]: pgmap v228: 161 pgs: 4 activating, 22 peering, 12 active+undersized+degraded, 16 active+undersized, 107 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 43/627 objects degraded (6.858%) 2024-02-10T06:31:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:30 smithi150 ceph-mon[68055]: Health check update: Degraded data redundancy: 43/627 objects degraded (6.858%), 12 pgs degraded (PG_DEGRADED) 2024-02-10T06:31:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:30 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:31.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:30 smithi139 ceph-mon[85480]: Health check update: Degraded data redundancy: 43/627 objects degraded (6.858%), 12 pgs degraded (PG_DEGRADED) 2024-02-10T06:31:31.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:30 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:31.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:31.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:30 smithi139 ceph-mon[87870]: Health check update: Degraded data redundancy: 43/627 objects degraded (6.858%), 12 pgs degraded (PG_DEGRADED) 2024-02-10T06:31:31.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:30 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:31.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:31.861 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:31 smithi150 ceph-mon[68055]: pgmap v229: 161 pgs: 4 activating, 22 peering, 9 active+undersized+degraded, 11 active+undersized, 115 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-02-10T06:31:32.070 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:31 smithi139 ceph-mon[85480]: pgmap v229: 161 pgs: 4 activating, 22 peering, 9 active+undersized+degraded, 11 active+undersized, 115 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-02-10T06:31:32.070 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:31 smithi139 ceph-mon[87870]: pgmap v229: 161 pgs: 4 activating, 22 peering, 9 active+undersized+degraded, 11 active+undersized, 115 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-02-10T06:31:32.336 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:31:32.069+0000 7ff625973700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T06:31:33.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:31:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[85480]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:33.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:33.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:32 smithi139 ceph-mon[87870]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:32 smithi150 ceph-mon[68055]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T06:31:34.238 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:33 smithi139 ceph-mon[85480]: pgmap v230: 161 pgs: 4 activating, 8 peering, 149 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 445 B/s rd, 0 op/s 2024-02-10T06:31:34.238 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:33 smithi139 ceph-mon[87870]: pgmap v230: 161 pgs: 4 activating, 8 peering, 149 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 445 B/s rd, 0 op/s 2024-02-10T06:31:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:33 smithi150 ceph-mon[68055]: pgmap v230: 161 pgs: 4 activating, 8 peering, 149 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 445 B/s rd, 0 op/s 2024-02-10T06:31:35.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:34 smithi139 ceph-mon[85480]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 9 pgs degraded) 2024-02-10T06:31:35.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:34 smithi139 ceph-mon[85480]: Cluster is now healthy 2024-02-10T06:31:35.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:34 smithi139 ceph-mon[87870]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 9 pgs degraded) 2024-02-10T06:31:35.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:34 smithi139 ceph-mon[87870]: Cluster is now healthy 2024-02-10T06:31:35.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:34 smithi150 ceph-mon[68055]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 9 pgs degraded) 2024-02-10T06:31:35.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:34 smithi150 ceph-mon[68055]: Cluster is now healthy 2024-02-10T06:31:35.503 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:31:36.144 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:31:36.144 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (8m) 112s ago 30m 26.3M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:31:36.144 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (8m) 8s ago 30m 86.5M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:31:36.144 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (8m) 112s ago 29m 64.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:31:36.144 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (9m) 8s ago 35m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:31:36.144 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (19m) 112s ago 36m 508M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:31:36.144 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (6m) 112s ago 36m 54.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (7m) 8s ago 35m 49.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (6m) 112s ago 35m 41.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (9m) 112s ago 31m 17.0M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (9m) 8s ago 31m 16.4M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (4m) 112s ago 34m 53.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (3m) 112s ago 34m 51.5M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (2m) 112s ago 34m 40.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (117s) 112s ago 33m 12.6M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e1826d3dcabe 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (82s) 8s ago 33m 52.8M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 f61046f26c97 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (48s) 8s ago 32m 42.5M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 24270c0f3351 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (13s) 8s ago 32m 12.6M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 6136d2cdfcc2 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (32m) 8s ago 32m 65.8M 3248M 17.2.0 e1d6a67b021e 6a90ad35933f 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (8m) 8s ago 31m 65.9M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (30m) 112s ago 29m 78.2M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:31:36.145 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (29m) 8s ago 29m 78.2M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:31:36.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:35 smithi139 ceph-mon[85480]: from='client.54573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:36.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:35 smithi139 ceph-mon[85480]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:36.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:35 smithi139 ceph-mon[87870]: from='client.54573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:36.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:35 smithi139 ceph-mon[87870]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:36.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:35 smithi150 ceph-mon[68055]: from='client.54573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:36.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:35 smithi150 ceph-mon[68055]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:36.530 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:31:36.530 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:31:36.530 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:31:36.530 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:31:36.530 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:31:36.530 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:31:36.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:31:36.531 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:31:36.531 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T06:31:36.531 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 7 2024-02-10T06:31:36.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:31:36.531 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:31:36.531 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:31:36.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:31:36.531 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:31:36.531 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-10T06:31:36.531 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 12 2024-02-10T06:31:36.532 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:31:36.532 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:31:36.858 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:31:36.859 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:31:36.859 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:31:36.859 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-10T06:31:36.859 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:31:36.859 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "7/8 daemons upgraded", 2024-02-10T06:31:36.859 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:31:36.859 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:31:36.859 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:31:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:36 smithi139 ceph-mon[85480]: from='client.54579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:37.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:36 smithi139 ceph-mon[85480]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:37.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:36 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/2784116608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:37.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:36 smithi139 ceph-mon[87870]: from='client.54579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:37.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:36 smithi139 ceph-mon[87870]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:37.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:36 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/2784116608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:37.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:36 smithi150 ceph-mon[68055]: from='client.54579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:37.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:36 smithi150 ceph-mon[68055]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:37.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:36 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/2784116608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:31:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:38 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:38 smithi139 ceph-mon[85480]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:38 smithi139 ceph-mon[85480]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:31:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:38 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:38 smithi139 ceph-mon[87870]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:38 smithi139 ceph-mon[87870]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:31:38.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:38 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:38 smithi150 ceph-mon[68055]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:31:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:38 smithi150 ceph-mon[68055]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:31:39.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:31:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:31:38] "GET /metrics HTTP/1.1" 200 38077 "" "Prometheus/2.43.0" 2024-02-10T06:31:39.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:39 smithi139 ceph-mon[87870]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-02-10T06:31:39.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:39 smithi139 ceph-mon[85480]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-02-10T06:31:39.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:39 smithi150 ceph-mon[68055]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-02-10T06:31:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:41 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:41 smithi150 ceph-mon[68055]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-02-10T06:31:42.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:41 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:42.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:41 smithi139 ceph-mon[85480]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-02-10T06:31:42.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:41 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:42.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:41 smithi139 ceph-mon[87870]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-02-10T06:31:43.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:43 smithi139 ceph-mon[85480]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-02-10T06:31:43.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:43 smithi139 ceph-mon[87870]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-02-10T06:31:43.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:43 smithi150 ceph-mon[68055]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-02-10T06:31:46.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:46 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:46.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:46 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:46.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:46 smithi139 ceph-mon[85480]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-10T06:31:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:46 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:46 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:46 smithi139 ceph-mon[87870]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-10T06:31:46.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:46 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:46 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:46 smithi150 ceph-mon[68055]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-10T06:31:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:48 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:48 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:48 smithi139 ceph-mon[87870]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:48 smithi139 ceph-mon[87870]: Upgrade: osd.7 is safe to restart 2024-02-10T06:31:48.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:48 smithi139 ceph-mon[87870]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:31:48.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:48 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:48.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:48 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:48.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:48 smithi139 ceph-mon[85480]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:48.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:48 smithi139 ceph-mon[85480]: Upgrade: osd.7 is safe to restart 2024-02-10T06:31:48.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:48 smithi139 ceph-mon[85480]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:31:48.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:48 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:48 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:48 smithi150 ceph-mon[68055]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:31:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:48 smithi150 ceph-mon[68055]: Upgrade: osd.7 is safe to restart 2024-02-10T06:31:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:48 smithi150 ceph-mon[68055]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:31:49.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:31:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:31:48] "GET /metrics HTTP/1.1" 200 38224 "" "Prometheus/2.43.0" 2024-02-10T06:31:49.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:49 smithi150 ceph-mon[68055]: Upgrade: Updating osd.7 2024-02-10T06:31:49.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:49 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:49.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:31:49.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:49.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:49 smithi150 ceph-mon[68055]: Deploying daemon osd.7 on smithi150 2024-02-10T06:31:49.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:49 smithi150 ceph-mon[68055]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:31:50.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[87870]: Upgrade: Updating osd.7 2024-02-10T06:31:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:31:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[87870]: Deploying daemon osd.7 on smithi150 2024-02-10T06:31:50.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[87870]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:31:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[85480]: Upgrade: Updating osd.7 2024-02-10T06:31:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:31:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[85480]: Deploying daemon osd.7 on smithi150 2024-02-10T06:31:50.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:49 smithi139 ceph-mon[85480]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-10T06:31:51.848 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:51 smithi150 ceph-mon[68055]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-02-10T06:31:52.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:51 smithi139 ceph-mon[85480]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-02-10T06:31:52.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:51 smithi139 ceph-mon[87870]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-02-10T06:31:52.121 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:51 smithi150 systemd[1]: Stopping Ceph osd.7 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:31:52.374 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[37768]: 2024-02-10T06:31:52.120+0000 7f14b7c5d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:31:52.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[37768]: 2024-02-10T06:31:52.120+0000 7f14b7c5d700 -1 osd.7 149 *** Got signal Terminated *** 2024-02-10T06:31:52.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[37768]: 2024-02-10T06:31:52.120+0000 7f14b7c5d700 -1 osd.7 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:31:53.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:53 smithi139 ceph-mon[85480]: osd.7 marked itself down and dead 2024-02-10T06:31:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:53 smithi139 ceph-mon[87870]: osd.7 marked itself down and dead 2024-02-10T06:31:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:53 smithi150 ceph-mon[68055]: osd.7 marked itself down and dead 2024-02-10T06:31:53.875 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:53 smithi150 podman[92834]: 2024-02-10 06:31:53.576163173 +0000 UTC m=+1.572322336 container died 6a90ad35933fb8f91c519b1c61914adde9b4087ba206da2830d11eeed0ec1c02 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, version=8, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , ceph=True, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, GIT_CLEAN=True, release=754, RELEASE=HEAD) 2024-02-10T06:31:54.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:54 smithi150 ceph-mon[68055]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:31:54.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:54 smithi150 ceph-mon[68055]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-10T06:31:54.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:54 smithi150 ceph-mon[68055]: osdmap e150: 8 total, 7 up, 8 in 2024-02-10T06:31:54.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:54 smithi150 ceph-mon[68055]: pgmap v241: 161 pgs: 11 peering, 13 stale+active+clean, 137 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T06:31:54.289 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:54 smithi150 podman[92834]: 2024-02-10 06:31:54.037264568 +0000 UTC m=+2.033423738 container cleanup 6a90ad35933fb8f91c519b1c61914adde9b4087ba206da2830d11eeed0ec1c02 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, RELEASE=HEAD, com.redhat.component=centos-stream-container, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , architecture=x86_64, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, version=8, name=centos-stream) 2024-02-10T06:31:54.289 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:54 smithi150 bash[92834]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7 2024-02-10T06:31:54.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:54 smithi139 ceph-mon[85480]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:31:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:54 smithi139 ceph-mon[85480]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-10T06:31:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:54 smithi139 ceph-mon[85480]: osdmap e150: 8 total, 7 up, 8 in 2024-02-10T06:31:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:54 smithi139 ceph-mon[85480]: pgmap v241: 161 pgs: 11 peering, 13 stale+active+clean, 137 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T06:31:54.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:54 smithi139 ceph-mon[87870]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:31:54.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:54 smithi139 ceph-mon[87870]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-10T06:31:54.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:54 smithi139 ceph-mon[87870]: osdmap e150: 8 total, 7 up, 8 in 2024-02-10T06:31:54.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:54 smithi139 ceph-mon[87870]: pgmap v241: 161 pgs: 11 peering, 13 stale+active+clean, 137 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T06:31:54.589 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:54 smithi150 podman[92862]: 2024-02-10 06:31:54.28855672 +0000 UTC m=+0.711676775 container remove 6a90ad35933fb8f91c519b1c61914adde9b4087ba206da2830d11eeed0ec1c02 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, ceph=True, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, distribution-scope=public) 2024-02-10T06:31:54.875 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:54 smithi150 podman[92958]: 2024-02-10 06:31:54.589637052 +0000 UTC m=+0.105203031 container create 473401fcb42093053fb40ee7031cc13df8741d2315cb1731a732ebf200d13413 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:31:54.875 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:54 smithi150 podman[92958]: 2024-02-10 06:31:54.503195752 +0000 UTC m=+0.018761693 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:31:54.875 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:54 smithi150 podman[92958]: 2024-02-10 06:31:54.808867436 +0000 UTC m=+0.324433368 container init 473401fcb42093053fb40ee7031cc13df8741d2315cb1731a732ebf200d13413 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T06:31:54.875 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:54 smithi150 podman[92958]: 2024-02-10 06:31:54.811857456 +0000 UTC m=+0.327423389 container start 473401fcb42093053fb40ee7031cc13df8741d2315cb1731a732ebf200d13413 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_CLEAN=True) 2024-02-10T06:31:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:55 smithi139 ceph-mon[85480]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-10T06:31:55.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:55 smithi139 ceph-mon[85480]: osdmap e151: 8 total, 7 up, 8 in 2024-02-10T06:31:55.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:55 smithi139 ceph-mon[87870]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-10T06:31:55.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:55 smithi139 ceph-mon[87870]: osdmap e151: 8 total, 7 up, 8 in 2024-02-10T06:31:55.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:55 smithi150 ceph-mon[68055]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-10T06:31:55.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:55 smithi150 ceph-mon[68055]: osdmap e151: 8 total, 7 up, 8 in 2024-02-10T06:31:55.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:54 smithi150 podman[92958]: 2024-02-10 06:31:54.892132709 +0000 UTC m=+0.407698660 container attach 473401fcb42093053fb40ee7031cc13df8741d2315cb1731a732ebf200d13413 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-02-10T06:31:55.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:55 smithi150 podman[92958]: 2024-02-10 06:31:55.113030905 +0000 UTC m=+0.628596850 container died 473401fcb42093053fb40ee7031cc13df8741d2315cb1731a732ebf200d13413 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:31:56.072 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:55 smithi150 podman[92984]: 2024-02-10 06:31:55.729163779 +0000 UTC m=+0.610711876 container remove 473401fcb42093053fb40ee7031cc13df8741d2315cb1731a732ebf200d13413 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-10T06:31:56.072 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:55 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.7.service: Deactivated successfully. 2024-02-10T06:31:56.072 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:55 smithi150 systemd[1]: Stopped Ceph osd.7 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:31:56.073 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:55 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.7.service: Consumed 17.392s CPU time. 2024-02-10T06:31:56.073 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:55 smithi150 systemd[1]: Starting Ceph osd.7 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:31:56.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:56 smithi139 ceph-mon[85480]: pgmap v243: 161 pgs: 42 peering, 7 stale+active+clean, 112 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:31:56.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:56 smithi139 ceph-mon[87870]: pgmap v243: 161 pgs: 42 peering, 7 stale+active+clean, 112 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:31:56.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:56 smithi150 ceph-mon[68055]: pgmap v243: 161 pgs: 42 peering, 7 stale+active+clean, 112 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:31:56.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 podman[93091]: 2024-02-10 06:31:56.180597842 +0000 UTC m=+0.114005418 container create 850f82ba11eec1aee3996c7157b4cc4a381a20babc64f3cdb10c39689420673e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-10T06:31:56.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 podman[93091]: 2024-02-10 06:31:56.087616542 +0000 UTC m=+0.021024084 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:31:56.842 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 podman[93091]: 2024-02-10 06:31:56.391644394 +0000 UTC m=+0.325051927 container init 850f82ba11eec1aee3996c7157b4cc4a381a20babc64f3cdb10c39689420673e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.vendor=CentOS, ceph=True) 2024-02-10T06:31:56.842 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 podman[93091]: 2024-02-10 06:31:56.394679566 +0000 UTC m=+0.328087103 container start 850f82ba11eec1aee3996c7157b4cc4a381a20babc64f3cdb10c39689420673e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-10T06:31:56.843 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 podman[93091]: 2024-02-10 06:31:56.466447592 +0000 UTC m=+0.399855129 container attach 850f82ba11eec1aee3996c7157b4cc4a381a20babc64f3cdb10c39689420673e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate[93106]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate[93106]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 bash[93091]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 bash[93091]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 bash[93091]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 bash[93091]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 bash[93091]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 bash[93091]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 bash[93091]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate[93106]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate[93106]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T06:31:57.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate[93106]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-10T06:31:57.126 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate[93106]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T06:31:57.126 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate[93106]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-10T06:31:57.126 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:56 smithi150 podman[93091]: 2024-02-10 06:31:56.865772357 +0000 UTC m=+0.799179891 container died 850f82ba11eec1aee3996c7157b4cc4a381a20babc64f3cdb10c39689420673e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-10T06:31:57.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:57 smithi150 podman[93238]: 2024-02-10 06:31:57.428537366 +0000 UTC m=+0.556518289 container remove 850f82ba11eec1aee3996c7157b4cc4a381a20babc64f3cdb10c39689420673e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2) 2024-02-10T06:31:57.906 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:57 smithi150 podman[93304]: 2024-02-10 06:31:57.548342963 +0000 UTC m=+0.018853884 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:31:57.906 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:57 smithi150 podman[93304]: 2024-02-10 06:31:57.679810408 +0000 UTC m=+0.150321321 container create 7818cf0fd8b527d604c9416794bd669711162dac69de53a158cc6d621476d9be (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-02-10T06:31:57.906 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:57 smithi150 podman[93304]: 2024-02-10 06:31:57.882264956 +0000 UTC m=+0.352775868 container init 7818cf0fd8b527d604c9416794bd669711162dac69de53a158cc6d621476d9be (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:31:57.906 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:57 smithi150 podman[93304]: 2024-02-10 06:31:57.885471966 +0000 UTC m=+0.355982879 container start 7818cf0fd8b527d604c9416794bd669711162dac69de53a158cc6d621476d9be (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, ceph=True, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T06:31:58.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:58 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:58.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:58 smithi139 ceph-mon[87870]: pgmap v244: 161 pgs: 6 active+undersized+degraded, 42 peering, 14 active+undersized, 99 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 17/627 objects degraded (2.711%) 2024-02-10T06:31:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:58 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:58 smithi139 ceph-mon[85480]: pgmap v244: 161 pgs: 6 active+undersized+degraded, 42 peering, 14 active+undersized, 99 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 17/627 objects degraded (2.711%) 2024-02-10T06:31:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:58.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:57 smithi150 ceph-osd[93323]: -- 172.21.15.150:0/2603368830 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55574dc111e0 con 0x55574dbe3400 2024-02-10T06:31:58.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:57 smithi150 bash[93304]: 7818cf0fd8b527d604c9416794bd669711162dac69de53a158cc6d621476d9be 2024-02-10T06:31:58.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:57 smithi150 systemd[1]: Started Ceph osd.7 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:31:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:58 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:58 smithi150 ceph-mon[68055]: pgmap v244: 161 pgs: 6 active+undersized+degraded, 42 peering, 14 active+undersized, 99 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 17/627 objects degraded (2.711%) 2024-02-10T06:31:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:31:58.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:58.874 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:31:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:31:58.430+0000 7fa137baa7c0 -1 Falling back to public interface 2024-02-10T06:31:59.065 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:31:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:31:58] "GET /metrics HTTP/1.1" 200 38224 "" "Prometheus/2.43.0" 2024-02-10T06:31:59.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:31:59 smithi139 ceph-mon[85480]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-02-10T06:31:59.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:31:59 smithi139 ceph-mon[87870]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-02-10T06:31:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:31:59 smithi150 ceph-mon[68055]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-02-10T06:32:00.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:00 smithi139 ceph-mon[85480]: pgmap v245: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T06:32:00.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:00 smithi139 ceph-mon[87870]: pgmap v245: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T06:32:00.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:00 smithi150 ceph-mon[68055]: pgmap v245: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T06:32:01.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:01 smithi139 ceph-mon[85480]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 7 pgs peering) 2024-02-10T06:32:01.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:01.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:01 smithi139 ceph-mon[87870]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 7 pgs peering) 2024-02-10T06:32:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:01.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:01.360 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:01 smithi150 ceph-mon[68055]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 7 pgs peering) 2024-02-10T06:32:01.360 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:01.360 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:01.361 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:32:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:32:01.152+0000 7fa137baa7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-02-10T06:32:01.361 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:32:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:32:01.193+0000 7fa137baa7c0 -1 osd.7 149 log_to_monitors true 2024-02-10T06:32:02.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:02 smithi139 ceph-mon[85480]: from='osd.7 [v2:172.21.15.150:6824/984657199,v1:172.21.15.150:6825/984657199]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:32:02.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:02 smithi139 ceph-mon[85480]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:32:02.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:02 smithi139 ceph-mon[85480]: pgmap v246: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T06:32:02.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:02 smithi139 ceph-mon[87870]: from='osd.7 [v2:172.21.15.150:6824/984657199,v1:172.21.15.150:6825/984657199]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:32:02.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:02 smithi139 ceph-mon[87870]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:32:02.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:02 smithi139 ceph-mon[87870]: pgmap v246: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T06:32:02.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:02 smithi150 ceph-mon[68055]: from='osd.7 [v2:172.21.15.150:6824/984657199,v1:172.21.15.150:6825/984657199]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:32:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:02 smithi150 ceph-mon[68055]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:32:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:02 smithi150 ceph-mon[68055]: pgmap v246: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T06:32:02.874 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:32:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:32:02.535+0000 7fa124ffa700 -1 osd.7 149 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:32:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[85480]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T06:32:03.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[85480]: from='osd.7 [v2:172.21.15.150:6824/984657199,v1:172.21.15.150:6825/984657199]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:32:03.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[85480]: osdmap e152: 8 total, 7 up, 8 in 2024-02-10T06:32:03.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[85480]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:32:03.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:03.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:03.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[85480]: from='osd.7 ' entity='osd.7' 2024-02-10T06:32:03.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[87870]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T06:32:03.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[87870]: from='osd.7 [v2:172.21.15.150:6824/984657199,v1:172.21.15.150:6825/984657199]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:32:03.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[87870]: osdmap e152: 8 total, 7 up, 8 in 2024-02-10T06:32:03.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[87870]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:32:03.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:03.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:03.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:03 smithi139 ceph-mon[87870]: from='osd.7 ' entity='osd.7' 2024-02-10T06:32:03.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:03 smithi150 ceph-mon[68055]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T06:32:03.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:03 smithi150 ceph-mon[68055]: from='osd.7 [v2:172.21.15.150:6824/984657199,v1:172.21.15.150:6825/984657199]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:32:03.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:03 smithi150 ceph-mon[68055]: osdmap e152: 8 total, 7 up, 8 in 2024-02-10T06:32:03.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:03 smithi150 ceph-mon[68055]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-10T06:32:03.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:03 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:03.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:03 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:03.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:03 smithi150 ceph-mon[68055]: from='osd.7 ' entity='osd.7' 2024-02-10T06:32:04.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[85480]: pgmap v248: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 434 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-02-10T06:32:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[85480]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:32:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:32:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[85480]: osd.7 [v2:172.21.15.150:6824/984657199,v1:172.21.15.150:6825/984657199] boot 2024-02-10T06:32:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[85480]: osdmap e153: 8 total, 8 up, 8 in 2024-02-10T06:32:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:04.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:04.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[87870]: pgmap v248: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 434 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-02-10T06:32:04.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[87870]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:32:04.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:32:04.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[87870]: osd.7 [v2:172.21.15.150:6824/984657199,v1:172.21.15.150:6825/984657199] boot 2024-02-10T06:32:04.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[87870]: osdmap e153: 8 total, 8 up, 8 in 2024-02-10T06:32:04.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:04.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:04 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:04.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:04 smithi150 ceph-mon[68055]: pgmap v248: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 434 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-02-10T06:32:04.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:04 smithi150 ceph-mon[68055]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:32:04.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:04 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:32:04.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:04 smithi150 ceph-mon[68055]: osd.7 [v2:172.21.15.150:6824/984657199,v1:172.21.15.150:6825/984657199] boot 2024-02-10T06:32:04.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:04 smithi150 ceph-mon[68055]: osdmap e153: 8 total, 8 up, 8 in 2024-02-10T06:32:04.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:04.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:04 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:05.727 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:05 smithi150 ceph-mon[68055]: osdmap e154: 8 total, 8 up, 8 in 2024-02-10T06:32:05.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:05 smithi139 ceph-mon[85480]: osdmap e154: 8 total, 8 up, 8 in 2024-02-10T06:32:05.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:05 smithi139 ceph-mon[87870]: osdmap e154: 8 total, 8 up, 8 in 2024-02-10T06:32:06.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:06 smithi150 ceph-mon[68055]: pgmap v251: 161 pgs: 25 peering, 18 active+undersized+degraded, 22 active+undersized, 96 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/627 objects degraded (9.410%) 2024-02-10T06:32:06.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:06 smithi150 ceph-mon[68055]: Health check update: Degraded data redundancy: 59/627 objects degraded (9.410%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T06:32:06.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:06 smithi139 ceph-mon[85480]: pgmap v251: 161 pgs: 25 peering, 18 active+undersized+degraded, 22 active+undersized, 96 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/627 objects degraded (9.410%) 2024-02-10T06:32:06.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:06 smithi139 ceph-mon[85480]: Health check update: Degraded data redundancy: 59/627 objects degraded (9.410%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T06:32:06.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:06 smithi139 ceph-mon[87870]: pgmap v251: 161 pgs: 25 peering, 18 active+undersized+degraded, 22 active+undersized, 96 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/627 objects degraded (9.410%) 2024-02-10T06:32:06.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:06 smithi139 ceph-mon[87870]: Health check update: Degraded data redundancy: 59/627 objects degraded (9.410%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T06:32:07.226 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:32:07.870 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:32:07.870 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (9m) 2m ago 31m 26.3M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:32:07.870 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (8m) 5s ago 30m 88.9M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:32:07.870 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (9m) 2m ago 30m 64.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:32:07.870 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (10m) 5s ago 35m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:32:07.870 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (20m) 2m ago 37m 508M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:32:07.870 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (7m) 2m ago 37m 54.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:32:07.870 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (7m) 5s ago 35m 50.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (6m) 2m ago 36m 41.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (9m) 2m ago 31m 17.0M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (9m) 5s ago 31m 16.4M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (4m) 2m ago 35m 53.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (4m) 2m ago 34m 51.5M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (3m) 2m ago 34m 40.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (2m) 2m ago 34m 12.6M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e1826d3dcabe 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (114s) 5s ago 33m 55.7M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 f61046f26c97 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (80s) 5s ago 33m 44.1M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 24270c0f3351 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (45s) 5s ago 33m 44.2M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 6136d2cdfcc2 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (10s) 5s ago 32m 15.1M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 7818cf0fd8b5 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (9m) 5s ago 32m 65.9M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (30m) 2m ago 30m 78.2M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:32:07.871 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (30m) 5s ago 30m 78.6M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:32:08.297 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:32:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:32:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:32:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:32:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:32:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:32:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:32:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:32:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:32:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:32:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:32:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:32:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:32:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:32:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-10T06:32:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 13 2024-02-10T06:32:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:32:08.298 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all osd 2024-02-10T06:32:08.299 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T06:32:08.300 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.301 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all osd 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T06:32:08.302 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T06:32:08.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T06:32:08.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T06:32:08.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T06:32:08.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T06:32:08.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T06:32:08.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T06:32:08.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T06:32:08.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T06:32:08.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T06:32:08.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T06:32:08.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: Upgrade: Setting require_osd_release to 19 squid 2024-02-10T06:32:08.304 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T06:32:08.304 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T06:32:08.304 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: from='client.64546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:08.304 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[87870]: pgmap v252: 161 pgs: 25 peering, 5 active+undersized+degraded, 2 active+undersized, 129 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 13/627 objects degraded (2.073%) 2024-02-10T06:32:08.304 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T06:32:08.304 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T06:32:08.304 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T06:32:08.304 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T06:32:08.304 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T06:32:08.304 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T06:32:08.304 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T06:32:08.305 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T06:32:08.305 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T06:32:08.305 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T06:32:08.305 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: Upgrade: Setting require_osd_release to 19 squid 2024-02-10T06:32:08.305 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T06:32:08.305 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T06:32:08.305 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: from='client.64546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:08.305 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:07 smithi139 ceph-mon[85480]: pgmap v252: 161 pgs: 25 peering, 5 active+undersized+degraded, 2 active+undersized, 129 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 13/627 objects degraded (2.073%) 2024-02-10T06:32:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all osd 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T06:32:08.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T06:32:08.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T06:32:08.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T06:32:08.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T06:32:08.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T06:32:08.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T06:32:08.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T06:32:08.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T06:32:08.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: Upgrade: Setting require_osd_release to 19 squid 2024-02-10T06:32:08.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T06:32:08.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T06:32:08.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: from='client.64546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:08.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:07 smithi150 ceph-mon[68055]: pgmap v252: 161 pgs: 25 peering, 5 active+undersized+degraded, 2 active+undersized, 129 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 13/627 objects degraded (2.073%) 2024-02-10T06:32:08.646 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:32:08.646 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": null, 2024-02-10T06:32:08.646 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": false, 2024-02-10T06:32:08.646 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "", 2024-02-10T06:32:08.646 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:32:08.646 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": null, 2024-02-10T06:32:08.646 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "", 2024-02-10T06:32:08.646 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:32:08.646 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:32:08.945 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:32:08] "GET /metrics HTTP/1.1" 200 38305 "" "Prometheus/2.43.0" 2024-02-10T06:32:08.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:08.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='client.64558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:08.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-10T06:32:08.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-10T06:32:08.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: osdmap e155: 8 total, 8 up, 8 in 2024-02-10T06:32:08.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mds 2024-02-10T06:32:08.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all nfs 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.948 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: Upgrade: Finalizing container_image settings 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:32:08.949 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:08.950 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:08.951 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:08.952 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: Upgrade: Complete! 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:09.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:09.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/993798730' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:09.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:09.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:09.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='client.64558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: osdmap e155: 8 total, 8 up, 8 in 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mds 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all nfs 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: Upgrade: Finalizing container_image settings 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:32:09.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:32:09.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:32:09.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.343 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: Upgrade: Complete! 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/993798730' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:09.344 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:08 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='client.64558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-10T06:32:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-10T06:32:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: osdmap e155: 8 total, 8 up, 8 in 2024-02-10T06:32:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mds 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all nfs 2024-02-10T06:32:09.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: Upgrade: Finalizing container_image settings 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:32:09.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.379 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.379 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:32:09.379 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:32:09.379 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:32:09.379 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:32:09.379 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:32:09.379 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:32:09.379 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:32:09.379 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: Upgrade: Complete! 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/993798730' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:09.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:08 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:09 smithi139 ceph-mon[85480]: from='client.54633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:10.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:09 smithi139 ceph-mon[85480]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:32:10.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:09 smithi139 ceph-mon[87870]: from='client.54633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:10.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:09 smithi139 ceph-mon[87870]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:32:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:09 smithi150 ceph-mon[68055]: from='client.54633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:09 smithi150 ceph-mon[68055]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:32:11.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:10 smithi139 ceph-mon[85480]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/627 objects degraded (2.073%), 5 pgs degraded) 2024-02-10T06:32:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:10 smithi139 ceph-mon[85480]: Cluster is now healthy 2024-02-10T06:32:11.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:10 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:10 smithi139 ceph-mon[87870]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/627 objects degraded (2.073%), 5 pgs degraded) 2024-02-10T06:32:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:10 smithi139 ceph-mon[87870]: Cluster is now healthy 2024-02-10T06:32:11.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:10 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:10 smithi150 ceph-mon[68055]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/627 objects degraded (2.073%), 5 pgs degraded) 2024-02-10T06:32:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:10 smithi150 ceph-mon[68055]: Cluster is now healthy 2024-02-10T06:32:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:10 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:12.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:11 smithi139 ceph-mon[85480]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:32:12.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:11 smithi139 ceph-mon[87870]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:32:12.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:11 smithi150 ceph-mon[68055]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:32:13.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:13 smithi150 ceph-mon[68055]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-02-10T06:32:14.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:13 smithi139 ceph-mon[87870]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-02-10T06:32:14.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:13 smithi139 ceph-mon[85480]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-02-10T06:32:16.736 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:16.736 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:16 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:16.736 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:16 smithi139 ceph-mon[87870]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:16.736 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:16.737 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:16.737 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:16.737 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:16 smithi139 ceph-mon[85480]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:16.737 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:16 smithi150 ceph-mon[68055]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:18.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:18 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:18.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:18 smithi139 ceph-mon[85480]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:18 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:18 smithi139 ceph-mon[87870]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:18.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:18 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:18.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:18 smithi150 ceph-mon[68055]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:19.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:32:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:32:18] "GET /metrics HTTP/1.1" 200 38311 "" "Prometheus/2.43.0" 2024-02-10T06:32:19.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:19 smithi150 ceph-mon[68055]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:32:20.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:19 smithi139 ceph-mon[85480]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:32:20.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:19 smithi139 ceph-mon[87870]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:32:21.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:21 smithi150 ceph-mon[68055]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:22.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:21 smithi139 ceph-mon[85480]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:22.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:21 smithi139 ceph-mon[87870]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:23.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:23 smithi150 ceph-mon[68055]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:24.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:23 smithi139 ceph-mon[85480]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:24.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:23 smithi139 ceph-mon[87870]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:25.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:25 smithi150 ceph-mon[68055]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:26.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:25 smithi139 ceph-mon[85480]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:26.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:25 smithi139 ceph-mon[87870]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:28.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:28 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:28.338 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:28 smithi139 ceph-mon[85480]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:28.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:28 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:28.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:28 smithi139 ceph-mon[87870]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:28 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:28 smithi150 ceph-mon[68055]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:29.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:32:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:32:28] "GET /metrics HTTP/1.1" 200 38311 "" "Prometheus/2.43.0" 2024-02-10T06:32:29.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:29 smithi150 ceph-mon[68055]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:30.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:29 smithi139 ceph-mon[85480]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:30.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:29 smithi139 ceph-mon[87870]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:30.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:31.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:31.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:31 smithi150 ceph-mon[68055]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:32.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:31 smithi139 ceph-mon[85480]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:32.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:31 smithi139 ceph-mon[87870]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:33.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:33 smithi150 ceph-mon[68055]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:34.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:33 smithi139 ceph-mon[85480]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:34.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:33 smithi139 ceph-mon[87870]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:35.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:35 smithi150 ceph-mon[68055]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:35.888 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:35 smithi139 ceph-mon[85480]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:35.888 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:35 smithi139 ceph-mon[87870]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:38 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:38.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:38 smithi139 ceph-mon[85480]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:38 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:38 smithi139 ceph-mon[87870]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:38 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:38 smithi150 ceph-mon[68055]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:39.000 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:32:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:32:38] "GET /metrics HTTP/1.1" 200 38309 "" "Prometheus/2.43.0" 2024-02-10T06:32:39.693 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:32:39.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:39 smithi150 ceph-mon[68055]: from='client.54639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:39.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:39 smithi150 ceph-mon[68055]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:40.002 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:39 smithi139 ceph-mon[87870]: from='client.54639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:40.002 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:39 smithi139 ceph-mon[87870]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:40.002 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:39 smithi139 ceph-mon[85480]: from='client.54639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:40.002 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:39 smithi139 ceph-mon[85480]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:41.536 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:32:41.536 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (9m) 2m ago 31m 26.3M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:32:41.536 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (9m) 39s ago 31m 88.9M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:32:41.536 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (9m) 2m ago 30m 64.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:32:41.536 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (10m) 39s ago 36m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:32:41.536 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (20m) 2m ago 37m 508M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:32:41.536 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (7m) 2m ago 38m 54.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:32:41.536 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (8m) 39s ago 36m 50.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (7m) 2m ago 36m 41.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (10m) 2m ago 32m 17.0M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (10m) 39s ago 32m 16.4M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (5m) 2m ago 35m 53.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (4m) 2m ago 35m 51.5M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (3m) 2m ago 35m 40.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (3m) 2m ago 34m 12.6M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e1826d3dcabe 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (2m) 39s ago 34m 55.7M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 f61046f26c97 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (113s) 39s ago 34m 44.1M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 24270c0f3351 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (79s) 39s ago 33m 44.2M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 6136d2cdfcc2 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (43s) 39s ago 33m 15.1M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 7818cf0fd8b5 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (10m) 39s ago 32m 65.9M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (31m) 2m ago 31m 78.2M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:32:41.537 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (31m) 39s ago 31m 78.6M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:32:41.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:41 smithi139 ceph-mon[85480]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:41.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:41 smithi139 ceph-mon[85480]: from='client.64582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:41.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:41 smithi139 ceph-mon[87870]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:41.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:41 smithi139 ceph-mon[87870]: from='client.64582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:41.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:41 smithi150 ceph-mon[68055]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:41 smithi150 ceph-mon[68055]: from='client.64582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:42.131 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 1'"'"'' 2024-02-10T06:32:43.849 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:43 smithi139 ceph-mon[85480]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:43.850 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:43 smithi139 ceph-mon[87870]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:43.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:43 smithi150 ceph-mon[68055]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:44.248 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:32:44.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:44 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/30018104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:44.978 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:44 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/30018104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:44.979 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:44 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/30018104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:45.004 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.osd | keys'"'"' | grep $sha1' 2024-02-10T06:32:45.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:45 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:45.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:45 smithi150 ceph-mon[68055]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:45.960 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:45 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:45.960 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:45 smithi139 ceph-mon[85480]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:45.960 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:45 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:45.961 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:45 smithi139 ceph-mon[87870]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)" 2024-02-10T06:32:47.882 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade status' 2024-02-10T06:32:48.186 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:48 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:48.186 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:48 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/2891214693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:48.186 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:48 smithi139 ceph-mon[85480]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:48.186 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:48 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:48.186 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:48 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/2891214693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:48.186 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:48 smithi139 ceph-mon[87870]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:48.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:48 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:48 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/2891214693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:32:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:48 smithi150 ceph-mon[68055]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:48.741 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:32:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:32:48] "GET /metrics HTTP/1.1" 200 38305 "" "Prometheus/2.43.0" 2024-02-10T06:32:49.746 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:32:49.746 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": null, 2024-02-10T06:32:49.746 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": false, 2024-02-10T06:32:49.746 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "", 2024-02-10T06:32:49.746 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:32:49.746 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": null, 2024-02-10T06:32:49.746 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "", 2024-02-10T06:32:49.746 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:32:49.746 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:32:49.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:49 smithi150 ceph-mon[68055]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:50.047 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:49 smithi139 ceph-mon[85480]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:50.048 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:49 smithi139 ceph-mon[87870]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:50.415 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T06:32:50.724 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:50 smithi139 ceph-mon[85480]: from='client.54663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:50.724 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:50 smithi139 ceph-mon[87870]: from='client.54663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:50.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:50 smithi150 ceph-mon[68055]: from='client.54663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:51.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:51 smithi150 ceph-mon[68055]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:51.944 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:51 smithi139 ceph-mon[85480]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:51.944 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:51 smithi139 ceph-mon[87870]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:52.445 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_OK 2024-02-10T06:32:53.219 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo' 2024-02-10T06:32:53.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:53 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/250911842' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:32:53.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:53 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/250911842' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:32:53.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:53 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/250911842' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:32:54.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:54 smithi139 ceph-mon[87870]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:54.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:54 smithi139 ceph-mon[85480]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:54.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:54 smithi150 ceph-mon[68055]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:55 smithi150 ceph-mon[68055]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:55.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:55 smithi150 ceph-mon[68055]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:56.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:55 smithi139 ceph-mon[85480]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:56.075 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:55 smithi139 ceph-mon[85480]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:56.075 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:55 smithi139 ceph-mon[87870]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:32:56.075 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:55 smithi139 ceph-mon[87870]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:57.090 INFO:teuthology.orchestra.run.smithi139.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:32:57.764 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-10T06:32:58.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[85480]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:32:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[85480]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:32:58.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[85480]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[87870]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:32:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[87870]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:32:58.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-mon[87870]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:58 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:58 smithi150 ceph-mon[68055]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:32:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:32:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:58 smithi150 ceph-mon[68055]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:32:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:58 smithi150 ceph-mon[68055]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:59.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:32:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:32:58] "GET /metrics HTTP/1.1" 200 38305 "" "Prometheus/2.43.0" 2024-02-10T06:32:59.816 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:32:59 smithi139 ceph-mon[85480]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:59.816 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:32:59 smithi139 ceph-mon[87870]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:59.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:32:59 smithi150 ceph-mon[68055]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:59.969 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:33:00.630 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:33:00.630 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (9m) 3m ago 32m 26.3M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:33:00.630 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (9m) 58s ago 31m 88.9M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:33:00.630 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (10m) 3m ago 31m 64.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:33:00.630 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (11m) 58s ago 36m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:33:00.630 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (21m) 3m ago 38m 508M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:33:00.630 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (7m) 3m ago 38m 54.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:33:00.630 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (8m) 58s ago 36m 50.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:33:00.630 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (7m) 3m ago 36m 41.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:33:00.630 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (10m) 3m ago 32m 17.0M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:33:00.630 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (10m) 58s ago 32m 16.4M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:33:00.631 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (5m) 3m ago 36m 53.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:33:00.631 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (5m) 3m ago 35m 51.5M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:33:00.631 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (4m) 3m ago 35m 40.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:33:00.631 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (3m) 3m ago 35m 12.6M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e1826d3dcabe 2024-02-10T06:33:00.631 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (2m) 58s ago 34m 55.7M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 f61046f26c97 2024-02-10T06:33:00.631 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (2m) 58s ago 34m 44.1M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 24270c0f3351 2024-02-10T06:33:00.631 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (98s) 58s ago 33m 44.2M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 6136d2cdfcc2 2024-02-10T06:33:00.631 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (63s) 58s ago 33m 15.1M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 7818cf0fd8b5 2024-02-10T06:33:00.631 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (10m) 58s ago 32m 65.9M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:33:00.631 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (31m) 3m ago 31m 78.2M - 17.2.0 e1d6a67b021e 382a725df409 2024-02-10T06:33:00.631 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (31m) 58s ago 31m 78.6M - 17.2.0 e1d6a67b021e b07749255c44 2024-02-10T06:33:00.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:00 smithi139 ceph-mon[85480]: from='client.54681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:00.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:00 smithi139 ceph-mon[85480]: from='client.64624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:00.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:00 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:00 smithi139 ceph-mon[87870]: from='client.54681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:00 smithi139 ceph-mon[87870]: from='client.64624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:00.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:00 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:00.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:00 smithi150 ceph-mon[68055]: from='client.54681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:00.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:00 smithi150 ceph-mon[68055]: from='client.64624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:00.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:00 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:01.036 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:33:01.036 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:33:01.036 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:33:01.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:33:01.036 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:33:01.036 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:33:01.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:33:01.036 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:33:01.036 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:33:01.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:33:01.037 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:33:01.037 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:33:01.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:33:01.037 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:33:01.037 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-10T06:33:01.037 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 13 2024-02-10T06:33:01.037 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:33:01.037 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:33:01.413 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:33:01.414 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:33:01.414 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:33:01.414 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading daemons in service(s) rgw.foo", 2024-02-10T06:33:01.414 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:33:01.414 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "0/2 daemons upgraded", 2024-02-10T06:33:01.414 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading rgw daemons", 2024-02-10T06:33:01.414 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:33:01.414 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:33:01.929 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='client.64630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mgr 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mon 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all crash 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.930 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all osd 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mds 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/4151557252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: from='client.64642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[87870]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='client.64630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:33:01.931 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mgr 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mon 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all crash 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all osd 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mds 2024-02-10T06:33:01.932 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:01.933 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/4151557252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:01.933 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: from='client.64642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:01.933 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:01 smithi139 ceph-mon[85480]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='client.64630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:33:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:33:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mgr 2024-02-10T06:33:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mon 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all crash 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all osd 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mds 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/4151557252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: from='client.64642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:02.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:01 smithi150 ceph-mon[68055]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[85480]: Upgrade: Updating rgw.foo.smithi139.kpyuou (1/2) 2024-02-10T06:33:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[85480]: Deploying daemon rgw.foo.smithi139.kpyuou on smithi139 2024-02-10T06:33:03.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[85480]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[87870]: Upgrade: Updating rgw.foo.smithi139.kpyuou (1/2) 2024-02-10T06:33:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:03.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:03.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:03.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:03.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[87870]: Deploying daemon rgw.foo.smithi139.kpyuou on smithi139 2024-02-10T06:33:03.838 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:03 smithi139 ceph-mon[87870]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:03.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:03 smithi150 ceph-mon[68055]: Upgrade: Updating rgw.foo.smithi139.kpyuou (1/2) 2024-02-10T06:33:03.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:03 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:03.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:03 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:03.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:03 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi139.kpyuou", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:03.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:03 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:03.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:03 smithi150 ceph-mon[68055]: Deploying daemon rgw.foo.smithi139.kpyuou on smithi139 2024-02-10T06:33:03.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:03 smithi150 ceph-mon[68055]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:05.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:05 smithi139 ceph-mon[85480]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:05.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:05 smithi139 ceph-mon[87870]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:05.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:05 smithi150 ceph-mon[68055]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:08.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:07 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:08.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:08.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:07 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:08.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:07 smithi139 ceph-mon[85480]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 20 op/s 2024-02-10T06:33:08.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:07 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:08.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:08.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:07 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:08.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:07 smithi139 ceph-mon[87870]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 20 op/s 2024-02-10T06:33:08.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:07 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:07 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:08.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:07 smithi150 ceph-mon[68055]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 20 op/s 2024-02-10T06:33:09.087 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:33:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:33:08] "GET /metrics HTTP/1.1" 200 38309 "" "Prometheus/2.43.0" 2024-02-10T06:33:09.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:09 smithi139 ceph-mon[85480]: Upgrade: Updating rgw.foo.smithi150.subshb (2/2) 2024-02-10T06:33:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:09 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:09 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:09 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:09.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:09 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:09.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:09 smithi139 ceph-mon[87870]: Upgrade: Updating rgw.foo.smithi150.subshb (2/2) 2024-02-10T06:33:09.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:09 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:09.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:09 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:09.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:09 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:09.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:09 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:09.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:09 smithi150 ceph-mon[68055]: Upgrade: Updating rgw.foo.smithi150.subshb (2/2) 2024-02-10T06:33:09.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:09 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:09.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:09 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:09.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:09 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi150.subshb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:33:09.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:09 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:10.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:10 smithi139 ceph-mon[85480]: Deploying daemon rgw.foo.smithi150.subshb on smithi150 2024-02-10T06:33:10.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:10 smithi139 ceph-mon[85480]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 85 B/s wr, 71 op/s 2024-02-10T06:33:10.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:10 smithi139 ceph-mon[87870]: Deploying daemon rgw.foo.smithi150.subshb on smithi150 2024-02-10T06:33:10.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:10 smithi139 ceph-mon[87870]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 85 B/s wr, 71 op/s 2024-02-10T06:33:10.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:10 smithi150 ceph-mon[68055]: Deploying daemon rgw.foo.smithi150.subshb on smithi150 2024-02-10T06:33:10.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:10 smithi150 ceph-mon[68055]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 85 B/s wr, 71 op/s 2024-02-10T06:33:11.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:11 smithi150 ceph-mon[68055]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 85 B/s wr, 71 op/s 2024-02-10T06:33:11.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:11 smithi139 ceph-mon[85480]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 85 B/s wr, 71 op/s 2024-02-10T06:33:11.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:11 smithi139 ceph-mon[87870]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 85 B/s wr, 71 op/s 2024-02-10T06:33:13.407 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:13 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:13.407 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:13 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:13.407 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:13 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:13.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:13 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:13.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:13 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:13.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:13 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:13.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:13 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:13.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:13 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:13.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:13 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:14.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:14 smithi139 ceph-mon[85480]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 170 B/s wr, 108 op/s 2024-02-10T06:33:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:14 smithi139 ceph-mon[87870]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 170 B/s wr, 108 op/s 2024-02-10T06:33:14.591 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:14 smithi150 ceph-mon[68055]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 170 B/s wr, 108 op/s 2024-02-10T06:33:15.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:15 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:15.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:15 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:15.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:15 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:16.780 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:16 smithi139 ceph-mon[85480]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 170 B/s wr, 135 op/s 2024-02-10T06:33:16.780 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:16.780 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:16.780 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:16.780 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:16 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:16.780 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:16 smithi139 ceph-mon[87870]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 170 B/s wr, 135 op/s 2024-02-10T06:33:16.780 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:16.781 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:16.781 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:16.781 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:16 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:16 smithi150 ceph-mon[68055]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 170 B/s wr, 135 op/s 2024-02-10T06:33:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:16 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.087 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[85480]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T06:33:18.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:18.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:17 smithi139 ceph-mon[87870]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T06:33:18.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:17 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:18.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:17 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:17 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:17 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:17 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:18.125 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:17 smithi150 ceph-mon[68055]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T06:33:18.863 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:33:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:33:18] "GET /metrics HTTP/1.1" 200 38309 "" "Prometheus/2.43.0" 2024-02-10T06:33:19.014 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:33:18 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a[66444]: logger=cleanup t=2024-02-10T06:33:18.738765678Z level=info msg="Completed cleanup jobs" duration=1.273038ms 2024-02-10T06:33:19.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:19 smithi139 ceph-mon[85480]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 170 B/s wr, 148 op/s 2024-02-10T06:33:19.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:19 smithi139 ceph-mon[87870]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 170 B/s wr, 148 op/s 2024-02-10T06:33:19.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:19 smithi150 ceph-mon[68055]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 170 B/s wr, 148 op/s 2024-02-10T06:33:21.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi139.kpyuou"}]: dispatch 2024-02-10T06:33:21.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi139.kpyuou"}]: dispatch 2024-02-10T06:33:21.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi139.kpyuou"}]': finished 2024-02-10T06:33:21.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi150.subshb"}]: dispatch 2024-02-10T06:33:21.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi150.subshb"}]: dispatch 2024-02-10T06:33:21.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi150.subshb"}]': finished 2024-02-10T06:33:21.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.839 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.841 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.842 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:21.842 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:21.842 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.842 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:21.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.843 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi139.kpyuou"}]: dispatch 2024-02-10T06:33:21.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi139.kpyuou"}]: dispatch 2024-02-10T06:33:21.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi139.kpyuou"}]': finished 2024-02-10T06:33:21.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi150.subshb"}]: dispatch 2024-02-10T06:33:21.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi150.subshb"}]: dispatch 2024-02-10T06:33:21.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi150.subshb"}]': finished 2024-02-10T06:33:21.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.844 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.845 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:33:21.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:33:21.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:33:21.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.846 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:33:21.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:33:21.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:33:21.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:33:21.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:33:21.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:33:21.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T06:33:21.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:33:21.847 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:33:21.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:33:21.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:33:21.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:33:21.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-10T06:33:21.848 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:33:21.849 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.849 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.849 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.849 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.849 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.849 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.849 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:33:21.849 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:33:21.849 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:33:21.850 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.850 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.850 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:33:21.850 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:33:21.850 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:33:21.850 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:33:21.850 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:33:21.850 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:33:21.850 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T06:33:21.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:33:21.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:33:21.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:33:21.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:33:21.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:33:21.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-10T06:33:21.851 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:33:21.852 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:33:21.852 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:33:21.852 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:33:21.854 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.855 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.856 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.856 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.856 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.856 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.856 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.856 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.856 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.856 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:21.857 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:21.858 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:21 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi139.kpyuou"}]: dispatch 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi139.kpyuou"}]: dispatch 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi139.kpyuou"}]': finished 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi150.subshb"}]: dispatch 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi150.subshb"}]: dispatch 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi150.subshb"}]': finished 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-10T06:33:21.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:33:21.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:33:21.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:33:21.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:33:21.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:33:21.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:33:21.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:21.881 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:21 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:22.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all rgw 2024-02-10T06:33:22.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:33:22.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:33:22.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all nfs 2024-02-10T06:33:22.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:33:22.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[87870]: Upgrade: Finalizing container_image settings 2024-02-10T06:33:22.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[87870]: Upgrade: Complete! 2024-02-10T06:33:22.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[87870]: Checking dashboard <-> RGW credentials 2024-02-10T06:33:22.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[87870]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 97 op/s 2024-02-10T06:33:22.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all rgw 2024-02-10T06:33:22.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:33:22.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:33:22.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all nfs 2024-02-10T06:33:22.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:33:22.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[85480]: Upgrade: Finalizing container_image settings 2024-02-10T06:33:22.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[85480]: Upgrade: Complete! 2024-02-10T06:33:22.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[85480]: Checking dashboard <-> RGW credentials 2024-02-10T06:33:22.838 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:22 smithi139 ceph-mon[85480]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 97 op/s 2024-02-10T06:33:22.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:22 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all rgw 2024-02-10T06:33:22.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:22 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:33:22.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:22 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:33:22.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:22 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all nfs 2024-02-10T06:33:22.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:22 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:33:22.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:22 smithi150 ceph-mon[68055]: Upgrade: Finalizing container_image settings 2024-02-10T06:33:22.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:22 smithi150 ceph-mon[68055]: Upgrade: Complete! 2024-02-10T06:33:22.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:22 smithi150 ceph-mon[68055]: Checking dashboard <-> RGW credentials 2024-02-10T06:33:22.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:22 smithi150 ceph-mon[68055]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 97 op/s 2024-02-10T06:33:23.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:23 smithi150 ceph-mon[68055]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 85 B/s wr, 99 op/s 2024-02-10T06:33:24.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:23 smithi139 ceph-mon[85480]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 85 B/s wr, 99 op/s 2024-02-10T06:33:24.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:23 smithi139 ceph-mon[87870]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 85 B/s wr, 99 op/s 2024-02-10T06:33:25.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:25 smithi150 ceph-mon[68055]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 62 op/s 2024-02-10T06:33:26.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:25 smithi139 ceph-mon[85480]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 62 op/s 2024-02-10T06:33:26.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:25 smithi139 ceph-mon[87870]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 62 op/s 2024-02-10T06:33:27.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:26 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:27.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:26 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:27.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:26 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:28 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:28.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:28 smithi150 ceph-mon[68055]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 36 op/s 2024-02-10T06:33:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:28 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:28.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:28 smithi139 ceph-mon[85480]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 36 op/s 2024-02-10T06:33:28.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:28 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:28.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:28 smithi139 ceph-mon[87870]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 36 op/s 2024-02-10T06:33:29.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:33:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:33:28] "GET /metrics HTTP/1.1" 200 38309 "" "Prometheus/2.43.0" 2024-02-10T06:33:29.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:29 smithi150 ceph-mon[68055]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-02-10T06:33:30.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:29 smithi139 ceph-mon[85480]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-02-10T06:33:30.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:29 smithi139 ceph-mon[87870]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-02-10T06:33:30.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:30 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:30.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:30 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:30.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:30 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:31.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:31 smithi150 ceph-mon[68055]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-10T06:33:32.068 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:31 smithi139 ceph-mon[85480]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-10T06:33:32.068 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:31 smithi139 ceph-mon[87870]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-10T06:33:32.493 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:33:33.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:33 smithi139 ceph-mon[87870]: from='client.54786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:33.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:33 smithi139 ceph-mon[85480]: from='client.54786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:33.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:33 smithi150 ceph-mon[68055]: from='client.54786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:34.136 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:34 smithi139 ceph-mon[85480]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-10T06:33:34.136 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:34 smithi139 ceph-mon[87870]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-10T06:33:34.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:34 smithi150 ceph-mon[68055]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-10T06:33:34.698 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:33:34.698 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (10m) 18s ago 32m 26.5M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:33:34.698 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (10m) 18s ago 32m 89.0M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:33:34.698 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (10m) 18s ago 31m 64.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (11m) 18s ago 37m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (21m) 18s ago 38m 513M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (8m) 18s ago 38m 62.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (9m) 18s ago 37m 52.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (8m) 18s ago 37m 47.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (11m) 18s ago 33m 18.4M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (11m) 18s ago 33m 16.2M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (6m) 18s ago 36m 61.1M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (5m) 18s ago 36m 62.3M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (4m) 18s ago 35m 47.8M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (3m) 18s ago 35m 61.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e1826d3dcabe 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (3m) 18s ago 35m 61.5M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 f61046f26c97 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (2m) 18s ago 34m 47.6M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 24270c0f3351 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (2m) 18s ago 34m 47.1M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 6136d2cdfcc2 2024-02-10T06:33:34.699 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (97s) 18s ago 34m 57.6M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 7818cf0fd8b5 2024-02-10T06:33:34.700 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (10m) 18s ago 33m 70.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:33:34.700 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (28s) 18s ago 31m 76.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 c027018c30fc 2024-02-10T06:33:34.700 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (22s) 18s ago 31m 74.6M - 19.0.0-1361-g290212a1 c328ab9dcfc7 af17a44dda49 2024-02-10T06:33:35.450 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.rgw | length == 1'"'"'' 2024-02-10T06:33:35.758 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:35 smithi139 ceph-mon[85480]: from='client.64726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:35.758 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:35 smithi139 ceph-mon[85480]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:33:35.759 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:35 smithi139 ceph-mon[87870]: from='client.64726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:35.759 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:35 smithi139 ceph-mon[87870]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:33:35.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:35 smithi150 ceph-mon[68055]: from='client.64726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:35.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:35 smithi150 ceph-mon[68055]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:33:37.418 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:33:38.161 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.rgw | keys'"'"' | grep $sha1' 2024-02-10T06:33:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:38 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:38 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/2397504069' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:38.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:38 smithi139 ceph-mon[85480]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:38 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:38 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/2397504069' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:38.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:38 smithi139 ceph-mon[87870]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:38.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:38 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:38 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/2397504069' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:38 smithi150 ceph-mon[68055]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:38.809 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:33:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:33:38] "GET /metrics HTTP/1.1" 200 38343 "" "Prometheus/2.43.0" 2024-02-10T06:33:39.815 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:39 smithi139 ceph-mon[85480]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:39.816 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:39 smithi139 ceph-mon[87870]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:39.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:39 smithi150 ceph-mon[68055]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:40.334 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)" 2024-02-10T06:33:40.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:40 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/2569604689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:41.035 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:40 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/2569604689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:41.035 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:40 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/2569604689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:41.060 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade status' 2024-02-10T06:33:41.663 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:41 smithi139 ceph-mon[85480]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:41.664 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:41 smithi139 ceph-mon[87870]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:41.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:41 smithi150 ceph-mon[68055]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:43.214 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:33:43.215 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": null, 2024-02-10T06:33:43.215 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": false, 2024-02-10T06:33:43.215 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "", 2024-02-10T06:33:43.215 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [], 2024-02-10T06:33:43.215 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": null, 2024-02-10T06:33:43.215 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "", 2024-02-10T06:33:43.215 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:33:43.215 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:33:43.832 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:43 smithi139 ceph-mon[85480]: from='client.64744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:43.833 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:43 smithi139 ceph-mon[85480]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:43.833 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:43 smithi139 ceph-mon[87870]: from='client.64744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:43.833 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:43 smithi139 ceph-mon[87870]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:43.857 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T06:33:43.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:43 smithi150 ceph-mon[68055]: from='client.64744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:43.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:43 smithi150 ceph-mon[68055]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:45.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:45 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:45.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:45 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:45.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:45 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:45.927 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_OK 2024-02-10T06:33:46.554 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-10T06:33:46.809 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:46 smithi139 ceph-mon[85480]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:46.809 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:46 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/3347196051' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:33:46.810 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:46 smithi139 ceph-mon[87870]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:46.810 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:46 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/3347196051' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:33:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:46 smithi150 ceph-mon[68055]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:46 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/3347196051' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:33:48.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:48 smithi150 ceph-mon[68055]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:48 smithi150 ceph-mon[68055]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:48.532 INFO:teuthology.orchestra.run.smithi139.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:33:48.533 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:48 smithi139 ceph-mon[85480]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:48.533 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:48 smithi139 ceph-mon[85480]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:48.533 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:48 smithi139 ceph-mon[87870]: from='client.25552 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:48.533 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:48 smithi139 ceph-mon[87870]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:48.836 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:33:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:33:48] "GET /metrics HTTP/1.1" 200 38345 "" "Prometheus/2.43.0" 2024-02-10T06:33:49.137 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:33:49.147 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi139.front.sepia.ceph.com 2024-02-10T06:33:49.148 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-10T06:33:49.717 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[85480]: from='client.64756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:49.717 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[85480]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:33:49.717 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:49.717 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:49.717 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:49.717 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:49.717 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:49.717 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[87870]: from='client.64756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:49.718 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[87870]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:33:49.718 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:49.718 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:49.718 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:49.718 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:49.718 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:49 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:49.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:49 smithi150 ceph-mon[68055]: from='client.64756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:49 smithi150 ceph-mon[68055]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:33:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:49 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:49 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:49.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:49 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:50.553 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:50 smithi139 ceph-mon[85480]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:33:50.553 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:50 smithi139 ceph-mon[85480]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:50.553 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:50 smithi139 ceph-mon[87870]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:33:50.553 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:50 smithi139 ceph-mon[87870]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:50.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:50 smithi150 ceph-mon[68055]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:33:50.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:50 smithi150 ceph-mon[68055]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:51.255 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:33:51.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:51 smithi139 ceph-mon[85480]: from='client.54828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:51.701 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:51 smithi139 ceph-mon[85480]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:51.701 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:51 smithi139 ceph-mon[87870]: from='client.54828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:51.701 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:51 smithi139 ceph-mon[87870]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:51.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:51 smithi150 ceph-mon[68055]: from='client.54828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:51.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:51 smithi150 ceph-mon[68055]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:51.937 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:33:51.937 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (10m) 36s ago 32m 26.5M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:33:51.937 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (10m) 36s ago 32m 89.0M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:33:51.937 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (10m) 36s ago 32m 64.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:33:51.937 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (11m) 36s ago 37m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:33:51.937 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (22m) 36s ago 39m 513M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (8m) 36s ago 39m 62.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (9m) 36s ago 37m 52.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (8m) 36s ago 37m 47.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (11m) 36s ago 33m 18.4M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (11m) 36s ago 33m 16.2M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (6m) 36s ago 37m 61.1M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (5m) 36s ago 36m 62.3M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (5m) 36s ago 36m 47.8M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (4m) 36s ago 35m 61.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e1826d3dcabe 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (3m) 36s ago 35m 61.5M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 f61046f26c97 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (3m) 36s ago 35m 47.6M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 24270c0f3351 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (2m) 36s ago 34m 47.1M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 6136d2cdfcc2 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (114s) 36s ago 34m 57.6M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 7818cf0fd8b5 2024-02-10T06:33:51.938 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (11m) 36s ago 33m 70.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:33:51.939 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (45s) 36s ago 32m 76.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 c027018c30fc 2024-02-10T06:33:51.939 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (40s) 36s ago 32m 74.6M - 19.0.0-1361-g290212a1 c328ab9dcfc7 af17a44dda49 2024-02-10T06:33:52.368 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:33:52.368 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:33:52.368 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:33:52.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:33:52.368 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:33:52.368 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:33:52.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:33:52.368 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:33:52.368 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:33:52.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:33:52.369 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:33:52.369 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:33:52.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:33:52.369 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:33:52.369 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 15 2024-02-10T06:33:52.369 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:33:52.369 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:33:52.740 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [ 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: "osd", 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: "mon", 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr", 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw" 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "15/21 daemons upgraded", 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Currently upgrading iscsi daemons", 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": false 2024-02-10T06:33:52.741 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:33:53.141 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_OK 2024-02-10T06:33:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='client.64768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='client.54840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:33:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:33:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mgr 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mon 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all crash 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all osd 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all mds 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all rgw 2024-02-10T06:33:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:33:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:33:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:33:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.339 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/3241334227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.339 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='client.64768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='client.54840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mgr 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mon 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all crash 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.340 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all osd 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all mds 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all rgw 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:33:53.341 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.342 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:53 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/3241334227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='client.64768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:53.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='client.54840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:53.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: Upgrade: Target is version 19.0.0-1361-g290212a1 (squid) 2024-02-10T06:33:53.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:33:53.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mgr 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mon 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all crash 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all osd 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all mds 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all rgw 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:53.627 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:53 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/3241334227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:33:54.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[85480]: from='client.64786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:54.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/2066157101' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:33:54.489 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[85480]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:54.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:54.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:33:54.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:33:54.490 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:54.490 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[87870]: from='client.64786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:54.490 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/2066157101' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:33:54.490 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[87870]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:54.490 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:54.490 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:33:54.490 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:33:54.490 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:54 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:54.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:54 smithi150 ceph-mon[68055]: from='client.64786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:33:54.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:54 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/2066157101' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:33:54.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:54 smithi150 ceph-mon[68055]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:54.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:54 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:54.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:54 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:33:54.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:54 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi139.mtlwwr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T06:33:54.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:54 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:55.434 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:55 smithi139 ceph-mon[85480]: Upgrade: Updating iscsi.foo.smithi139.mtlwwr 2024-02-10T06:33:55.435 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:55 smithi139 ceph-mon[85480]: Deploying daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:33:55.435 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:55 smithi139 ceph-mon[87870]: Upgrade: Updating iscsi.foo.smithi139.mtlwwr 2024-02-10T06:33:55.435 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:55 smithi139 ceph-mon[87870]: Deploying daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:33:55.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:55 smithi150 ceph-mon[68055]: Upgrade: Updating iscsi.foo.smithi139.mtlwwr 2024-02-10T06:33:55.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:55 smithi150 ceph-mon[68055]: Deploying daemon iscsi.foo.smithi139.mtlwwr on smithi139 2024-02-10T06:33:56.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:56 smithi139 ceph-mon[85480]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:56.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:56 smithi139 ceph-mon[87870]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:56.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:56 smithi150 ceph-mon[68055]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:57.703 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:33:57 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:33:57.348+0000 7ff5f610b700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi139.mtlwwr on host smithi139 failed. 2024-02-10T06:33:58.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:58 smithi150 ceph-mon[68055]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi139.mtlwwr on host smithi139 failed. 2024-02-10T06:33:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:58 smithi150 ceph-mon[68055]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-02-10T06:33:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:58.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:58.675 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[85480]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi139.mtlwwr on host smithi139 failed. 2024-02-10T06:33:58.675 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:58.675 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:58.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[85480]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-02-10T06:33:58.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:58.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:58.676 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:58.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[87870]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi139.mtlwwr on host smithi139 failed. 2024-02-10T06:33:58.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:58.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:58.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[87870]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-02-10T06:33:58.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:58.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:58.676 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:33:59.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:33:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:33:58] "GET /metrics HTTP/1.1" 200 38345 "" "Prometheus/2.43.0" 2024-02-10T06:33:59.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:33:59 smithi150 ceph-mon[68055]: Health check failed: Upgrading daemon iscsi.foo.smithi139.mtlwwr on host smithi139 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-10T06:33:59.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:33:59 smithi139 ceph-mon[85480]: Health check failed: Upgrading daemon iscsi.foo.smithi139.mtlwwr on host smithi139 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-10T06:33:59.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:33:59 smithi139 ceph-mon[87870]: Health check failed: Upgrading daemon iscsi.foo.smithi139.mtlwwr on host smithi139 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-10T06:34:00.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:00 smithi150 ceph-mon[68055]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:00.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:00 smithi139 ceph-mon[85480]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:00.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:00 smithi139 ceph-mon[87870]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:01.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:01 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:34:01.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:01 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:34:01.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:01.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:01 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:34:01.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:02.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:02 smithi139 ceph-mon[85480]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-02-10T06:34:02.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:02 smithi139 ceph-mon[87870]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-02-10T06:34:02.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:02 smithi150 ceph-mon[68055]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-02-10T06:34:04.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:04 smithi139 ceph-mon[85480]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-02-10T06:34:04.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:04 smithi139 ceph-mon[87870]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-02-10T06:34:04.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:04 smithi150 ceph-mon[68055]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-02-10T06:34:06.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:06 smithi139 ceph-mon[85480]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-02-10T06:34:06.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:06 smithi139 ceph-mon[87870]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-02-10T06:34:06.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:06 smithi150 ceph-mon[68055]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 779 B/s rd, 0 op/s 2024-02-10T06:34:08.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:08 smithi139 ceph-mon[85480]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 346 B/s rd, 0 op/s 2024-02-10T06:34:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:08 smithi139 ceph-mon[87870]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 346 B/s rd, 0 op/s 2024-02-10T06:34:08.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:08 smithi150 ceph-mon[68055]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 346 B/s rd, 0 op/s 2024-02-10T06:34:09.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:34:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:34:08] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-02-10T06:34:09.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:09 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/464179696' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:34:09.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:09 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/1524189402' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3526415303"}]: dispatch 2024-02-10T06:34:09.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:09 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/464179696' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:34:09.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:09 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/1524189402' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3526415303"}]: dispatch 2024-02-10T06:34:09.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:09 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/464179696' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T06:34:09.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:09 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/1524189402' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3526415303"}]: dispatch 2024-02-10T06:34:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:10 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/1524189402' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3526415303"}]': finished 2024-02-10T06:34:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:10 smithi150 ceph-mon[68055]: osdmap e156: 8 total, 8 up, 8 in 2024-02-10T06:34:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:10 smithi150 ceph-mon[68055]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-10T06:34:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:10 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/1825593957' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1897410992"}]: dispatch 2024-02-10T06:34:10.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:10 smithi150 ceph-mon[68055]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1897410992"}]: dispatch 2024-02-10T06:34:10.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:10 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/1524189402' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3526415303"}]': finished 2024-02-10T06:34:10.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:10 smithi139 ceph-mon[85480]: osdmap e156: 8 total, 8 up, 8 in 2024-02-10T06:34:10.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:10 smithi139 ceph-mon[85480]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-10T06:34:10.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:10 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/1825593957' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1897410992"}]: dispatch 2024-02-10T06:34:10.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:10 smithi139 ceph-mon[85480]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1897410992"}]: dispatch 2024-02-10T06:34:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:10 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/1524189402' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3526415303"}]': finished 2024-02-10T06:34:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:10 smithi139 ceph-mon[87870]: osdmap e156: 8 total, 8 up, 8 in 2024-02-10T06:34:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:10 smithi139 ceph-mon[87870]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-10T06:34:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:10 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/1825593957' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1897410992"}]: dispatch 2024-02-10T06:34:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:10 smithi139 ceph-mon[87870]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1897410992"}]: dispatch 2024-02-10T06:34:11.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:11 smithi150 ceph-mon[68055]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1897410992"}]': finished 2024-02-10T06:34:11.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:11 smithi150 ceph-mon[68055]: osdmap e157: 8 total, 8 up, 8 in 2024-02-10T06:34:11.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:11 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/4023669504' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3370473998"}]: dispatch 2024-02-10T06:34:11.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:11 smithi150 ceph-mon[68055]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3370473998"}]: dispatch 2024-02-10T06:34:11.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:11 smithi139 ceph-mon[85480]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1897410992"}]': finished 2024-02-10T06:34:11.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:11 smithi139 ceph-mon[85480]: osdmap e157: 8 total, 8 up, 8 in 2024-02-10T06:34:11.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:11 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/4023669504' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3370473998"}]: dispatch 2024-02-10T06:34:11.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:11 smithi139 ceph-mon[85480]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3370473998"}]: dispatch 2024-02-10T06:34:11.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:11 smithi139 ceph-mon[87870]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/1897410992"}]': finished 2024-02-10T06:34:11.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:11 smithi139 ceph-mon[87870]: osdmap e157: 8 total, 8 up, 8 in 2024-02-10T06:34:11.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:11 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/4023669504' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3370473998"}]: dispatch 2024-02-10T06:34:11.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:11 smithi139 ceph-mon[87870]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3370473998"}]: dispatch 2024-02-10T06:34:12.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:12 smithi139 ceph-mon[85480]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3370473998"}]': finished 2024-02-10T06:34:12.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:12 smithi139 ceph-mon[85480]: osdmap e158: 8 total, 8 up, 8 in 2024-02-10T06:34:12.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:12 smithi139 ceph-mon[85480]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:34:12.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:12 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/551241667' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2811448709"}]: dispatch 2024-02-10T06:34:12.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:12 smithi139 ceph-mon[85480]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2811448709"}]: dispatch 2024-02-10T06:34:12.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:12 smithi139 ceph-mon[87870]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3370473998"}]': finished 2024-02-10T06:34:12.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:12 smithi139 ceph-mon[87870]: osdmap e158: 8 total, 8 up, 8 in 2024-02-10T06:34:12.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:12 smithi139 ceph-mon[87870]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:34:12.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:12 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/551241667' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2811448709"}]: dispatch 2024-02-10T06:34:12.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:12 smithi139 ceph-mon[87870]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2811448709"}]: dispatch 2024-02-10T06:34:12.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:12 smithi150 ceph-mon[68055]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3370473998"}]': finished 2024-02-10T06:34:12.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:12 smithi150 ceph-mon[68055]: osdmap e158: 8 total, 8 up, 8 in 2024-02-10T06:34:12.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:12 smithi150 ceph-mon[68055]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:34:12.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:12 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/551241667' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2811448709"}]: dispatch 2024-02-10T06:34:12.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:12 smithi150 ceph-mon[68055]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2811448709"}]: dispatch 2024-02-10T06:34:13.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:13 smithi139 ceph-mon[85480]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2811448709"}]': finished 2024-02-10T06:34:13.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:13 smithi139 ceph-mon[85480]: osdmap e159: 8 total, 8 up, 8 in 2024-02-10T06:34:13.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:13 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/3522662474' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3383690878"}]: dispatch 2024-02-10T06:34:13.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:13 smithi139 ceph-mon[87870]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2811448709"}]': finished 2024-02-10T06:34:13.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:13 smithi139 ceph-mon[87870]: osdmap e159: 8 total, 8 up, 8 in 2024-02-10T06:34:13.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:13 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/3522662474' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3383690878"}]: dispatch 2024-02-10T06:34:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:13 smithi150 ceph-mon[68055]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6800/2811448709"}]': finished 2024-02-10T06:34:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:13 smithi150 ceph-mon[68055]: osdmap e159: 8 total, 8 up, 8 in 2024-02-10T06:34:13.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:13 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/3522662474' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3383690878"}]: dispatch 2024-02-10T06:34:14.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:14 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/3522662474' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3383690878"}]': finished 2024-02-10T06:34:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:14 smithi139 ceph-mon[87870]: osdmap e160: 8 total, 8 up, 8 in 2024-02-10T06:34:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:14 smithi139 ceph-mon[87870]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:34:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:14 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/1404656926' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2811448709"}]: dispatch 2024-02-10T06:34:14.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:14 smithi139 ceph-mon[87870]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2811448709"}]: dispatch 2024-02-10T06:34:14.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:14 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/3522662474' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3383690878"}]': finished 2024-02-10T06:34:14.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:14 smithi139 ceph-mon[85480]: osdmap e160: 8 total, 8 up, 8 in 2024-02-10T06:34:14.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:14 smithi139 ceph-mon[85480]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:34:14.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:14 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/1404656926' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2811448709"}]: dispatch 2024-02-10T06:34:14.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:14 smithi139 ceph-mon[85480]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2811448709"}]: dispatch 2024-02-10T06:34:14.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:14 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/3522662474' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:0/3383690878"}]': finished 2024-02-10T06:34:14.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:14 smithi150 ceph-mon[68055]: osdmap e160: 8 total, 8 up, 8 in 2024-02-10T06:34:14.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:14 smithi150 ceph-mon[68055]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:34:14.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:14 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/1404656926' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2811448709"}]: dispatch 2024-02-10T06:34:14.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:14 smithi150 ceph-mon[68055]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2811448709"}]: dispatch 2024-02-10T06:34:15.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:15 smithi139 ceph-mon[85480]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2811448709"}]': finished 2024-02-10T06:34:15.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:15 smithi139 ceph-mon[85480]: osdmap e161: 8 total, 8 up, 8 in 2024-02-10T06:34:15.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:15 smithi139 ceph-mon[87870]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2811448709"}]': finished 2024-02-10T06:34:15.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:15 smithi139 ceph-mon[87870]: osdmap e161: 8 total, 8 up, 8 in 2024-02-10T06:34:15.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:15 smithi150 ceph-mon[68055]: from='client.? ' entity='client.iscsi.foo.smithi139.mtlwwr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.139:6801/2811448709"}]': finished 2024-02-10T06:34:15.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:15 smithi150 ceph-mon[68055]: osdmap e161: 8 total, 8 up, 8 in 2024-02-10T06:34:16.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:16 smithi139 ceph-mon[85480]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:16.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:16 smithi139 ceph-mon[87870]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:16.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:16 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:16.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:16 smithi150 ceph-mon[68055]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:16.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:18.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:18 smithi139 ceph-mon[85480]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:18.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:18 smithi139 ceph-mon[87870]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:18.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:18 smithi150 ceph-mon[68055]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:19.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:34:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:34:18] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-02-10T06:34:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:19 smithi150 ceph-mon[68055]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:19.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:19 smithi139 ceph-mon[85480]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:19.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:19 smithi139 ceph-mon[87870]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:20 smithi150 ceph-mon[68055]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:34:20.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:20 smithi139 ceph-mon[85480]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:34:20.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:20 smithi139 ceph-mon[87870]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:34:22.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:22 smithi139 ceph-mon[85480]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:22.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:22 smithi139 ceph-mon[87870]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:22.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:22 smithi150 ceph-mon[68055]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:23.498 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:34:23.826 INFO:teuthology.orchestra.run.smithi139.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi139.mtlwwr on host smithi139 failed." 2024-02-10T06:34:24.570 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:34:24.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:24 smithi139 ceph-mon[87870]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:34:24.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:24 smithi139 ceph-mon[87870]: from='client.54924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:24.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:24 smithi139 ceph-mon[85480]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:34:24.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:24 smithi139 ceph-mon[85480]: from='client.54924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:24 smithi150 ceph-mon[68055]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:34:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:24 smithi150 ceph-mon[68055]: from='client.54924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:25.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:25 smithi139 ceph-mon[85480]: from='client.64864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:25.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:25 smithi139 ceph-mon[87870]: from='client.64864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:25.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:25 smithi150 ceph-mon[68055]: from='client.64864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:26.546 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:34:26.546 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (11m) 70s ago 33m 26.5M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:34:26.546 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (11m) 70s ago 33m 89.0M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (11m) 70s ago 32m 64.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (12m) 70s ago 37m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (22m) 70s ago 39m 513M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (9m) 70s ago 39m 62.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (10m) 70s ago 38m 52.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (8m) 70s ago 38m 47.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (12m) 70s ago 34m 18.4M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (12m) 70s ago 34m 16.2M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (7m) 70s ago 37m 61.1M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (6m) 70s ago 37m 62.3M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (5m) 70s ago 36m 47.8M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (4m) 70s ago 36m 61.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e1826d3dcabe 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (4m) 70s ago 36m 61.5M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 f61046f26c97 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (3m) 70s ago 35m 47.6M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 24270c0f3351 2024-02-10T06:34:26.547 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (3m) 70s ago 35m 47.1M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 6136d2cdfcc2 2024-02-10T06:34:26.548 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (2m) 70s ago 35m 57.6M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 7818cf0fd8b5 2024-02-10T06:34:26.548 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (11m) 70s ago 34m 70.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:34:26.548 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (80s) 70s ago 32m 76.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 c027018c30fc 2024-02-10T06:34:26.548 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (74s) 70s ago 32m 74.6M - 19.0.0-1361-g290212a1 c328ab9dcfc7 af17a44dda49 2024-02-10T06:34:26.817 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:26 smithi139 ceph-mon[85480]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-02-10T06:34:26.817 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:26 smithi139 ceph-mon[87870]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-02-10T06:34:26.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:26 smithi150 ceph-mon[68055]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-02-10T06:34:27.125 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T06:34:27.689 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:27 smithi139 ceph-mon[85480]: from='client.54936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:27.689 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:27 smithi139 ceph-mon[87870]: from='client.54936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:27.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:27 smithi150 ceph-mon[68055]: from='client.54936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:34:28.674 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:28 smithi139 ceph-mon[85480]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:28.675 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:28 smithi139 ceph-mon[87870]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:28.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:28 smithi150 ceph-mon[68055]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:29.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:34:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:34:28] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-02-10T06:34:29.218 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:34:29.218 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:34:29.218 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:34:29.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:34:29.218 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:34:29.218 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:34:29.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:34:29.218 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:34:29.219 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:34:29.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:34:29.219 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:34:29.219 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:34:29.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:34:29.219 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:34:29.219 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 15 2024-02-10T06:34:29.219 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:34:29.219 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:34:29.581 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:29 smithi139 ceph-mon[85480]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:29.581 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:29 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/3930974252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:34:29.582 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:29 smithi139 ceph-mon[87870]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:29.582 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:29 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/3930974252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:34:29.839 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-02-10T06:34:29.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:29 smithi150 ceph-mon[68055]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:29.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:29 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/3930974252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:34:30.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:30 smithi139 ceph-mon[85480]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:30.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:30 smithi139 ceph-mon[87870]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:30.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:30 smithi150 ceph-mon[68055]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:31.658 INFO:teuthology.orchestra.run.smithi139.stdout:wait for servicemap items w/ changing names to refresh 2024-02-10T06:34:31.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:31 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:31.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:31 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:31.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:31 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:32.398 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'sleep 60' 2024-02-10T06:34:32.707 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:32 smithi139 ceph-mon[85480]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:32.708 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:32 smithi139 ceph-mon[87870]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:32.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:32 smithi150 ceph-mon[68055]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:34.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:34 smithi139 ceph-mon[85480]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:34.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:34 smithi139 ceph-mon[87870]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:34.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:34 smithi150 ceph-mon[68055]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:36.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:36 smithi139 ceph-mon[85480]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:36.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:36 smithi139 ceph-mon[87870]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:36.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:36 smithi150 ceph-mon[68055]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:38 smithi150 ceph-mon[68055]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:38.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:38 smithi139 ceph-mon[85480]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:38.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:38 smithi139 ceph-mon[87870]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:39.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:34:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:34:38] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-02-10T06:34:39.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:39 smithi150 ceph-mon[68055]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:39.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:39 smithi139 ceph-mon[85480]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:39.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:39 smithi139 ceph-mon[87870]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:40.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:40 smithi150 ceph-mon[68055]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:40.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:40 smithi139 ceph-mon[85480]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:40.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:40 smithi139 ceph-mon[87870]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:42.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:42 smithi139 ceph-mon[85480]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:42.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:42 smithi139 ceph-mon[87870]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:42.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:42 smithi150 ceph-mon[68055]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:44.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:44 smithi139 ceph-mon[85480]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:44.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:44 smithi139 ceph-mon[87870]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:44.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:44 smithi150 ceph-mon[68055]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:46.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:46 smithi139 ceph-mon[85480]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:46.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:46 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:46 smithi139 ceph-mon[87870]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:46.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:46 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:46 smithi150 ceph-mon[68055]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:46 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:48 smithi150 ceph-mon[68055]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:34:48.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:48 smithi139 ceph-mon[85480]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:34:48.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:48 smithi139 ceph-mon[87870]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:34:49.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:34:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:34:48] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-02-10T06:34:49.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:49 smithi150 ceph-mon[68055]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:49.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:49 smithi139 ceph-mon[85480]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:49.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:49 smithi139 ceph-mon[87870]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:50.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:50 smithi150 ceph-mon[68055]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:50.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:50 smithi139 ceph-mon[85480]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:50.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:50 smithi139 ceph-mon[87870]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:52.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:52 smithi139 ceph-mon[85480]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:34:52.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:52 smithi139 ceph-mon[87870]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:34:52.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:52 smithi150 ceph-mon[68055]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:34:54.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:54 smithi139 ceph-mon[85480]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:54.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:54 smithi139 ceph-mon[87870]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:54.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:54 smithi150 ceph-mon[68055]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:56.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:56 smithi139 ceph-mon[87870]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:56.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:56 smithi139 ceph-mon[85480]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:56.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:56 smithi150 ceph-mon[68055]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:58 smithi150 ceph-mon[68055]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:34:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:34:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:58 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:34:58.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:58 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:34:58.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:58 smithi139 ceph-mon[87870]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:58.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:34:58.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:34:58.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:58 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:34:58.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:58 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:34:58.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:58 smithi139 ceph-mon[85480]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:58.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:34:58.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:34:58.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:58 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:34:58.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:58 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:34:59.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:34:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:34:58] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-02-10T06:34:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:34:59 smithi150 ceph-mon[68055]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:59.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:34:59 smithi139 ceph-mon[85480]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:59.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:34:59 smithi139 ceph-mon[87870]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:00.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:00 smithi150 ceph-mon[68055]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:00.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:00 smithi139 ceph-mon[85480]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:00.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:00 smithi139 ceph-mon[87870]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:01.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:01.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:01.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:02.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:02 smithi150 ceph-mon[68055]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:02.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:02 smithi139 ceph-mon[85480]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:02.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:02 smithi139 ceph-mon[87870]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:04.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:04 smithi139 ceph-mon[85480]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:04.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:04 smithi139 ceph-mon[87870]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:04.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:04 smithi150 ceph-mon[68055]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:06.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:06 smithi139 ceph-mon[85480]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:06.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:06 smithi139 ceph-mon[87870]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:06.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:06 smithi150 ceph-mon[68055]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:08.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:08 smithi139 ceph-mon[85480]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:08.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:08 smithi139 ceph-mon[87870]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:08.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:08 smithi150 ceph-mon[68055]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:09.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:35:08 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:35:08] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-02-10T06:35:09.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:09 smithi150 ceph-mon[68055]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:09.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:09 smithi139 ceph-mon[85480]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:09.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:09 smithi139 ceph-mon[87870]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:10.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:10 smithi150 ceph-mon[68055]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:10.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:10 smithi139 ceph-mon[85480]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:10.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:10 smithi139 ceph-mon[87870]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:12.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:12 smithi139 ceph-mon[85480]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:35:12.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:12 smithi139 ceph-mon[87870]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:35:12.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:12 smithi150 ceph-mon[68055]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:35:14.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:14 smithi139 ceph-mon[85480]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:14.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:14 smithi139 ceph-mon[87870]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:14.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:14 smithi150 ceph-mon[68055]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:16.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:16 smithi139 ceph-mon[87870]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:16.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:16 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:16.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:16 smithi139 ceph-mon[85480]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:16.837 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:16 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:16.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:16 smithi150 ceph-mon[68055]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:16.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:16 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:18.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:18 smithi150 ceph-mon[68055]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:18.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:18 smithi139 ceph-mon[85480]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:18.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:18 smithi139 ceph-mon[87870]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:19.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:35:18 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:35:18] "GET /metrics HTTP/1.1" 200 38424 "" "Prometheus/2.43.0" 2024-02-10T06:35:19.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:19 smithi150 ceph-mon[68055]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:19.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:19 smithi139 ceph-mon[85480]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:19.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:19 smithi139 ceph-mon[87870]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:20 smithi150 ceph-mon[68055]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:20.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:20 smithi139 ceph-mon[85480]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:20.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:20 smithi139 ceph-mon[87870]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:22.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:22 smithi139 ceph-mon[85480]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:22.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:22 smithi139 ceph-mon[87870]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:22.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:22 smithi150 ceph-mon[68055]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:24.840 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:24 smithi139 ceph-mon[85480]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:24.840 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:24 smithi139 ceph-mon[87870]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:24.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:24 smithi150 ceph-mon[68055]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:26.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:26 smithi139 ceph-mon[85480]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:26.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:26 smithi139 ceph-mon[87870]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:26.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:26 smithi150 ceph-mon[68055]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:28.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:28 smithi150 ceph-mon[68055]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:28.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:28 smithi139 ceph-mon[85480]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:28.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:28 smithi139 ceph-mon[87870]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:29.088 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:35:28 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:35:28] "GET /metrics HTTP/1.1" 200 38424 "" "Prometheus/2.43.0" 2024-02-10T06:35:29.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:29 smithi150 ceph-mon[68055]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:29.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:29 smithi139 ceph-mon[85480]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:29.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:29 smithi139 ceph-mon[87870]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:30.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:30 smithi150 ceph-mon[68055]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:30.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:30 smithi139 ceph-mon[85480]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:30.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:30 smithi139 ceph-mon[87870]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:31.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:31 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:31.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:31 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:31.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:31 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:32.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:32 smithi139 ceph-mon[85480]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:32.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:32 smithi139 ceph-mon[87870]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:32.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:32 smithi150 ceph-mon[68055]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:34.836 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:34 smithi139 ceph-mon[85480]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:34.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:34 smithi139 ceph-mon[87870]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:34.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:34 smithi150 ceph-mon[68055]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:34.959 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:35:36.706 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:36 smithi139 ceph-mon[85480]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:36.706 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:36 smithi139 ceph-mon[87870]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:36.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:36 smithi150 ceph-mon[68055]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:36.935 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:35:36.935 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.a smithi139 *:9093,9094 running (12m) 2m ago 34m 26.5M - 0.25.0 c8568f914cd2 2b6a81902a4c 2024-02-10T06:35:36.935 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.a smithi150 *:3000 running (12m) 2m ago 34m 89.0M - 9.4.12 a0ea5e3c46d9 dae5565f9ce7 2024-02-10T06:35:36.935 INFO:teuthology.orchestra.run.smithi139.stdout:iscsi.foo.smithi139.mtlwwr smithi139 running (12m) 2m ago 33m 64.7M - 3.5 e1d6a67b021e 626e2de99564 2024-02-10T06:35:36.935 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.x smithi150 *:8443,9283,8765 running (13m) 2m ago 39m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 f312c83e6dd4 2024-02-10T06:35:36.935 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.y smithi139 *:8443,9283,8765 running (23m) 2m ago 40m 513M - 19.0.0-1361-g290212a1 c328ab9dcfc7 082caf310592 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:mon.a smithi139 running (10m) 2m ago 40m 62.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ee03df6a685e 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:mon.b smithi150 running (11m) 2m ago 39m 52.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ec59077c047e 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:mon.c smithi139 running (10m) 2m ago 39m 47.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2dd65b7a1ca6 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.a smithi139 *:9100 running (13m) 2m ago 35m 18.4M - 1.5.0 0da6a335fe13 3c4db3eedf5e 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.b smithi150 *:9100 running (13m) 2m ago 35m 16.2M - 1.5.0 0da6a335fe13 b250d2d9c43c 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (8m) 2m ago 38m 61.1M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8287393c3c4f 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (7m) 2m ago 38m 62.3M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e33be38d0736 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (6m) 2m ago 38m 47.8M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 8d3bec1b0040 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi139 running (5m) 2m ago 37m 61.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 e1826d3dcabe 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi150 running (5m) 2m ago 37m 61.5M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 f61046f26c97 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi150 running (4m) 2m ago 36m 47.6M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 24270c0f3351 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:osd.6 smithi150 running (4m) 2m ago 36m 47.1M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 6136d2cdfcc2 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:osd.7 smithi150 running (3m) 2m ago 36m 57.6M 3248M 19.0.0-1361-g290212a1 c328ab9dcfc7 7818cf0fd8b5 2024-02-10T06:35:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.a smithi150 *:9095 running (12m) 2m ago 35m 70.1M - 2.43.0 a07b618ecd1d 8836434edb60 2024-02-10T06:35:36.937 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi139.kpyuou smithi139 *:8000 running (2m) 2m ago 34m 76.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 c027018c30fc 2024-02-10T06:35:36.937 INFO:teuthology.orchestra.run.smithi139.stdout:rgw.foo.smithi150.subshb smithi150 *:8000 running (2m) 2m ago 33m 74.6M - 19.0.0-1361-g290212a1 c328ab9dcfc7 af17a44dda49 2024-02-10T06:35:37.549 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T06:35:38.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:38 smithi150 ceph-mon[68055]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:38.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:38 smithi150 ceph-mon[68055]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:38.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:38 smithi139 ceph-mon[87870]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:38.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:38 smithi139 ceph-mon[87870]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:38.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:38 smithi139 ceph-mon[85480]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:38.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:38 smithi139 ceph-mon[85480]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:38.974 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:35:38 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:35:38] "GET /metrics HTTP/1.1" 200 38419 "" "Prometheus/2.43.0" 2024-02-10T06:35:39.276 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:39 smithi139 ceph-mon[85480]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:39.276 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:39 smithi139 ceph-mon[87870]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:39.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:39 smithi150 ceph-mon[68055]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:39.762 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:35:39.762 INFO:teuthology.orchestra.run.smithi139.stdout: "mon": { 2024-02-10T06:35:39.762 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:35:39.762 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:35:39.762 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr": { 2024-02-10T06:35:39.762 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:35:39.763 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:35:39.763 INFO:teuthology.orchestra.run.smithi139.stdout: "osd": { 2024-02-10T06:35:39.763 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:35:39.763 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:35:39.763 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw": { 2024-02-10T06:35:39.763 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:35:39.763 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-02-10T06:35:39.763 INFO:teuthology.orchestra.run.smithi139.stdout: "overall": { 2024-02-10T06:35:39.763 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 15 2024-02-10T06:35:39.763 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-02-10T06:35:39.763 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:35:40.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:40 smithi139 ceph-mon[87870]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:40.336 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:40 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/3007615285' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:35:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:40 smithi139 ceph-mon[85480]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:40.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:40 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/3007615285' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:35:40.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:40 smithi150 ceph-mon[68055]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:40.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:40 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/3007615285' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:35:40.381 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade status' 2024-02-10T06:35:42.335 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:35:42.335 INFO:teuthology.orchestra.run.smithi139.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:35:42.335 INFO:teuthology.orchestra.run.smithi139.stdout: "in_progress": true, 2024-02-10T06:35:42.335 INFO:teuthology.orchestra.run.smithi139.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:35:42.335 INFO:teuthology.orchestra.run.smithi139.stdout: "services_complete": [ 2024-02-10T06:35:42.335 INFO:teuthology.orchestra.run.smithi139.stdout: "osd", 2024-02-10T06:35:42.335 INFO:teuthology.orchestra.run.smithi139.stdout: "mon", 2024-02-10T06:35:42.336 INFO:teuthology.orchestra.run.smithi139.stdout: "mgr", 2024-02-10T06:35:42.336 INFO:teuthology.orchestra.run.smithi139.stdout: "rgw" 2024-02-10T06:35:42.336 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-02-10T06:35:42.336 INFO:teuthology.orchestra.run.smithi139.stdout: "progress": "15/21 daemons upgraded", 2024-02-10T06:35:42.336 INFO:teuthology.orchestra.run.smithi139.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi139.mtlwwr on host smithi139 failed.", 2024-02-10T06:35:42.336 INFO:teuthology.orchestra.run.smithi139.stdout: "is_paused": true 2024-02-10T06:35:42.336 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:35:42.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:42 smithi139 ceph-mon[85480]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:42.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:42 smithi139 ceph-mon[87870]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:42.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:42 smithi150 ceph-mon[68055]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:42.986 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T06:35:43.587 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:43 smithi139 ceph-mon[85480]: from='client.44723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:43.588 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:43 smithi139 ceph-mon[87870]: from='client.44723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:43.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:43 smithi150 ceph-mon[68055]: from='client.44723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:44.480 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:44 smithi139 ceph-mon[85480]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:44.480 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:44 smithi139 ceph-mon[87870]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:44.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:44 smithi150 ceph-mon[68055]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:44.955 INFO:teuthology.orchestra.run.smithi139.stdout:HEALTH_WARN Upgrading daemon iscsi.foo.smithi139.mtlwwr on host smithi139 failed. 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi139.mtlwwr on host smithi139 failed. 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrade daemon: iscsi.foo.smithi139.mtlwwr: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi139.mtlwwr ... 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout:Creating ceph-iscsi config... 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout:Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/iscsi-gateway.cfg 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout:Write file: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/iscsi.foo.smithi139.mtlwwr/tcmu-runner-entrypoint.sh 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2dce44e742\x2dc7d8\x2d11ee\x2d95b8\x2d87774f69a715.slice/ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout:Non-zero exit code 1 from systemctl start ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout:systemctl: stderr Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout:systemctl: stderr See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout:Traceback (most recent call last): 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout: return _run_code(code, main_globals, None, 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T06:35:44.956 INFO:teuthology.orchestra.run.smithi139.stdout: exec(code, run_globals) 2024-02-10T06:35:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T06:35:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T06:35:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T06:35:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T06:35:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T06:35:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1029, in deploy_daemon 2024-02-10T06:35:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1165, in deploy_daemon_units 2024-02-10T06:35:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: File "/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T06:35:44.957 INFO:teuthology.orchestra.run.smithi139.stdout:RuntimeError: Failed command: systemctl start ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr: Job for ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service failed because the control process exited with error code. 2024-02-10T06:35:44.957 INFO:teuthology.orchestra.run.smithi139.stdout:See "systemctl status ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" and "journalctl -xeu ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@iscsi.foo.smithi139.mtlwwr.service" for details. 2024-02-10T06:35:45.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:45 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/3887313422' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:35:45.586 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:45 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/3887313422' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:35:45.638 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-02-10T06:35:45.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:45 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/3887313422' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:35:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:46 smithi139 ceph-mon[85480]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:46.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:46 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:46.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:46 smithi139 ceph-mon[87870]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:46.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:46 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:46 smithi150 ceph-mon[68055]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:46.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:46 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:35:47.630 INFO:teuthology.orchestra.run.smithi139.stdout:true 2024-02-10T06:35:48.176 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:48 smithi139 ceph-mon[85480]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:48.176 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:48 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/1116784830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:35:48.176 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:48 smithi139 ceph-mon[87870]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:48.176 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:48 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/1116784830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:35:48.230 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-02-10T06:35:48.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:48 smithi150 ceph-mon[68055]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:48.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:48 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/1116784830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:35:48.787 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:35:48 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:35:48] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-02-10T06:35:49.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:49 smithi150 ceph-mon[68055]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:49.534 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:49 smithi139 ceph-mon[85480]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:49.535 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:49 smithi139 ceph-mon[87870]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:50.281 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)" 2024-02-10T06:35:50.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:50 smithi139 ceph-mon[85480]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:50.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:50 smithi139 ceph-mon[87870]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:50.625 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:50 smithi150 ceph-mon[68055]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:50.909 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-02-10T06:35:51.217 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:51 smithi139 ceph-mon[85480]: from='client.? 172.21.15.139:0/2497682323' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:35:51.217 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:51 smithi139 ceph-mon[87870]: from='client.? 172.21.15.139:0/2497682323' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:35:51.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:51 smithi150 ceph-mon[68055]: from='client.? 172.21.15.139:0/2497682323' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:35:52.180 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:52 smithi139 ceph-mon[85480]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:52.180 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:52 smithi139 ceph-mon[87870]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:52.527 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:52 smithi150 ceph-mon[68055]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:52.899 INFO:teuthology.orchestra.run.smithi139.stdout:osd 8 2m ago - 2024-02-10T06:35:53.527 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:35:53.538 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi139.front.sepia.ceph.com 2024-02-10T06:35:53.538 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-02-10T06:35:54.810 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:54 smithi139 ceph-mon[85480]: from='client.54984 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:54.810 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:54 smithi139 ceph-mon[85480]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:54.811 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:54 smithi139 ceph-mon[87870]: from='client.54984 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:54.811 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:54 smithi139 ceph-mon[87870]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:54.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:54 smithi150 ceph-mon[68055]: from='client.54984 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:54.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:54 smithi150 ceph-mon[68055]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:55.901 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-02-10T06:35:55.901 INFO:teuthology.orchestra.run.smithi139.stdout: "image": "quay.io/ceph/ceph", 2024-02-10T06:35:55.901 INFO:teuthology.orchestra.run.smithi139.stdout: "registry": "quay.io", 2024-02-10T06:35:55.901 INFO:teuthology.orchestra.run.smithi139.stdout: "bare_image": "ceph/ceph", 2024-02-10T06:35:55.901 INFO:teuthology.orchestra.run.smithi139.stdout: "versions": [] 2024-02-10T06:35:55.901 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-02-10T06:35:56.424 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-02-10T06:35:56.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:56 smithi139 ceph-mon[85480]: from='client.54990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:56.736 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:56 smithi139 ceph-mon[85480]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:56.737 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:56 smithi139 ceph-mon[87870]: from='client.54990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:56.737 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:56 smithi139 ceph-mon[87870]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:56.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:56 smithi150 ceph-mon[68055]: from='client.54990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:56.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:56 smithi150 ceph-mon[68055]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:35:58.336 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:58 smithi139 ceph-mon[85480]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:58.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:58 smithi139 ceph-mon[87870]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:58.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:58 smithi150 ceph-mon[68055]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:35:59.062 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:35:58 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: ::ffff:172.21.15.150 - - [10/Feb/2024:06:35:58] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-02-10T06:35:59.062 INFO:teuthology.orchestra.run.smithi139.stdout: "16.2.0", 2024-02-10T06:35:59.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:35:59.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[85480]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:59.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[85480]: from='client.64930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:59.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:35:59.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:35:59.337 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[85480]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:35:59.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:35:59.337 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[87870]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:59.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[87870]: from='client.64930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:59.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:35:59.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:35:59.338 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:35:59 smithi139 ceph-mon[87870]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:35:59.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:59 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:35:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:59 smithi150 ceph-mon[68055]: from='client.54873 -' entity='client.iscsi.foo.smithi139.mtlwwr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:35:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:59 smithi150 ceph-mon[68055]: from='client.64930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:35:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:59 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:35:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:59 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:35:59.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:35:59 smithi150 ceph-mon[68055]: from='mgr.25756 ' entity='mgr.y' 2024-02-10T06:35:59.686 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-02-10T06:36:00.281 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:36:00 smithi139 ceph-mon[85480]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:36:00.282 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:36:00 smithi139 ceph-mon[87870]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:36:00.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:00 smithi150 ceph-mon[68055]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:36:01.151 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:36:01 smithi139 ceph-mon[85480]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:36:01.151 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:36:01 smithi139 ceph-mon[87870]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:36:01.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:01 smithi150 ceph-mon[68055]: from='mgr.25756 172.21.15.139:0/1026068808' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:36:02.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:36:02 smithi139 ceph-mon[85480]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:36:02.587 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:36:02 smithi139 ceph-mon[87870]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:36:02.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:02 smithi150 ceph-mon[68055]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:36:02.665 INFO:teuthology.orchestra.run.smithi139.stdout: "v16.2.2", 2024-02-10T06:36:02.665 INFO:teuthology.orchestra.run.smithi139.stdout: "v16.2.2-20210505", 2024-02-10T06:36:03.266 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:36:03 smithi139 ceph-mon[85480]: from='client.55002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:36:03.267 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:36:03 smithi139 ceph-mon[87870]: from='client.55002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:36:03.295 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T06:36:03.305 INFO:tasks.cephadm:Teardown begin 2024-02-10T06:36:03.306 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:36:03.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:36:03.375 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T06:36:03.375 DEBUG:teuthology.orchestra.run.smithi139:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T06:36:03.396 DEBUG:teuthology.orchestra.run.smithi150:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T06:36:03.433 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T06:36:03.433 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-10T06:36:03.433 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.a 2024-02-10T06:36:03.586 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:36:03 smithi139 systemd[1]: Stopping Ceph mon.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:03.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:03 smithi150 ceph-mon[68055]: from='client.55002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:36:04.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:36:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a[85476]: 2024-02-10T06:36:03.727+0000 7f144c6dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:04.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:36:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a[85476]: 2024-02-10T06:36:03.727+0000 7f144c6dc700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-02-10T06:36:04.086 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:36:03 smithi139 podman[129564]: 2024-02-10 06:36:03.921793044 +0000 UTC m=+0.294869177 container died ee03df6a685e43fe9b26357bff470e61f785fef52f608e587f6ac23189a18aa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_BRANCH=HEAD) 2024-02-10T06:36:04.553 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:36:04 smithi139 podman[129564]: 2024-02-10 06:36:04.285609908 +0000 UTC m=+0.658686086 container cleanup ee03df6a685e43fe9b26357bff470e61f785fef52f608e587f6ac23189a18aa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:36:04.553 INFO:journalctl@ceph.mon.a.smithi139.stdout:Feb 10 06:36:04 smithi139 bash[129564]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-a 2024-02-10T06:36:04.697 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.a.service' 2024-02-10T06:36:04.749 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:36:04.749 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-10T06:36:04.750 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-10T06:36:04.750 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.c 2024-02-10T06:36:04.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:36:04 smithi139 systemd[1]: Stopping Ceph mon.c for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:05.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:36:05 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c[87866]: 2024-02-10T06:36:05.047+0000 7f276f3a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:05.303 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:36:05 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c[87866]: 2024-02-10T06:36:05.047+0000 7f276f3a0700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-02-10T06:36:05.574 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:36:05 smithi139 podman[129688]: 2024-02-10 06:36:05.303531226 +0000 UTC m=+0.368062283 container died 2dd65b7a1ca643f2008e2af53d7f004ed545c2b702c95d0379e35947ca92c22e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_CLEAN=True) 2024-02-10T06:36:05.836 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:36:05 smithi139 podman[129688]: 2024-02-10 06:36:05.672770565 +0000 UTC m=+0.737301628 container cleanup 2dd65b7a1ca643f2008e2af53d7f004ed545c2b702c95d0379e35947ca92c22e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T06:36:05.837 INFO:journalctl@ceph.mon.c.smithi139.stdout:Feb 10 06:36:05 smithi139 bash[129688]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-c 2024-02-10T06:36:06.140 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.c.service' 2024-02-10T06:36:06.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:36:06.176 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-10T06:36:06.176 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-10T06:36:06.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.b 2024-02-10T06:36:06.548 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:06 smithi150 systemd[1]: Stopping Ceph mon.b for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:06.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b[68051]: 2024-02-10T06:36:06.546+0000 7f21a1002700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b[68051]: 2024-02-10T06:36:06.546+0000 7f21a1002700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-02-10T06:36:06.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:06 smithi150 podman[101907]: 2024-02-10 06:36:06.629160759 +0000 UTC m=+0.257443397 container died ec59077c047e2307507159003ccdb94a9ba436e549d632b318e582e84428ac72 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-10T06:36:07.258 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:07 smithi150 podman[101907]: 2024-02-10 06:36:07.081069544 +0000 UTC m=+0.709352182 container cleanup ec59077c047e2307507159003ccdb94a9ba436e549d632b318e582e84428ac72 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.license=GPLv2) 2024-02-10T06:36:07.258 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:07 smithi150 bash[101907]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b 2024-02-10T06:36:07.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.b.service' 2024-02-10T06:36:07.518 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:07 smithi150 podman[101921]: 2024-02-10 06:36:07.257940498 +0000 UTC m=+0.619364876 container remove ec59077c047e2307507159003ccdb94a9ba436e549d632b318e582e84428ac72 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mon-b, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-290212a, GIT_CLEAN=True) 2024-02-10T06:36:07.518 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:07 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.b.service: Deactivated successfully. 2024-02-10T06:36:07.518 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:07 smithi150 systemd[1]: Stopped Ceph mon.b for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:36:07.518 INFO:journalctl@ceph.mon.b.smithi150.stdout:Feb 10 06:36:07 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mon.b.service: Consumed 9.689s CPU time. 2024-02-10T06:36:07.547 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:36:07.547 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-10T06:36:07.547 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-10T06:36:07.547 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.y 2024-02-10T06:36:07.786 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:36:07 smithi139 systemd[1]: Stopping Ceph mgr.y for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:08.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:36:07 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:36:07.867+0000 7ff664195700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:08.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:36:07 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y[63873]: 2024-02-10T06:36:07.867+0000 7ff664195700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-10T06:36:08.086 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:36:07 smithi139 podman[129812]: 2024-02-10 06:36:07.909536486 +0000 UTC m=+0.164907117 container died 082caf31059201e756d7ca81a17910710cb11ea888413e1679ee6203e5a95cb9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True) 2024-02-10T06:36:08.428 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:36:08 smithi139 podman[129812]: 2024-02-10 06:36:08.193257869 +0000 UTC m=+0.448628506 container cleanup 082caf31059201e756d7ca81a17910710cb11ea888413e1679ee6203e5a95cb9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a) 2024-02-10T06:36:08.428 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Feb 10 06:36:08 smithi139 bash[129812]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-y 2024-02-10T06:36:08.575 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.y.service' 2024-02-10T06:36:08.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:36:08.616 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-10T06:36:08.616 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-10T06:36:08.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.x 2024-02-10T06:36:08.958 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:36:08 smithi150 systemd[1]: Stopping Ceph mgr.x for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:09.236 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:36:08 smithi150 podman[102030]: 2024-02-10 06:36:08.958419 +0000 UTC m=+0.151744223 container died f312c83e6dd43d1da1111ae653c6fcab28bfd4ab0b594320bba90f1d17c1112c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-10T06:36:09.590 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:36:09 smithi150 podman[102030]: 2024-02-10 06:36:09.235951382 +0000 UTC m=+0.429276611 container cleanup f312c83e6dd43d1da1111ae653c6fcab28bfd4ab0b594320bba90f1d17c1112c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a) 2024-02-10T06:36:09.590 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:36:09 smithi150 bash[102030]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x 2024-02-10T06:36:09.590 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:36:09 smithi150 podman[102044]: 2024-02-10 06:36:09.462259663 +0000 UTC m=+0.501999683 container remove f312c83e6dd43d1da1111ae653c6fcab28bfd4ab0b594320bba90f1d17c1112c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-mgr-x, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-10T06:36:09.590 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Feb 10 06:36:09 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:36:09.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@mgr.x.service' 2024-02-10T06:36:09.640 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:36:09.641 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-10T06:36:09.641 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-10T06:36:09.641 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.0 2024-02-10T06:36:09.948 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:09 smithi139 systemd[1]: Stopping Ceph osd.0 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:10.336 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[95528]: 2024-02-10T06:36:09.946+0000 7f136c258700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:10.337 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[95528]: 2024-02-10T06:36:09.946+0000 7f136c258700 -1 osd.0 161 *** Got signal Terminated *** 2024-02-10T06:36:10.337 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:09 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0[95528]: 2024-02-10T06:36:09.946+0000 7f136c258700 -1 osd.0 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:36:15.631 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:15 smithi139 podman[129941]: 2024-02-10 06:36:15.349308223 +0000 UTC m=+5.506371308 container died 8287393c3c4f1b22711aec71463b2a9a45ffcbf65302025284a83ae1cdce53de (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a) 2024-02-10T06:36:15.998 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:15 smithi139 podman[129941]: 2024-02-10 06:36:15.730142313 +0000 UTC m=+5.887205398 container cleanup 8287393c3c4f1b22711aec71463b2a9a45ffcbf65302025284a83ae1cdce53de (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=main-290212a, io.buildah.version=1.33.2) 2024-02-10T06:36:15.998 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:15 smithi139 bash[129941]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0 2024-02-10T06:36:16.336 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:15 smithi139 podman[129958]: 2024-02-10 06:36:15.998349606 +0000 UTC m=+0.648591149 container remove 8287393c3c4f1b22711aec71463b2a9a45ffcbf65302025284a83ae1cdce53de (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0, RELEASE=main-290212a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS) 2024-02-10T06:36:16.627 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:16 smithi139 podman[130048]: 2024-02-10 06:36:16.248913498 +0000 UTC m=+0.019027280 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:36:16.627 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:16 smithi139 podman[130048]: 2024-02-10 06:36:16.357378405 +0000 UTC m=+0.127492173 container create f7d93cf586b1e5c671501068e3568534f98bbda21f94b75063c26f05aa0312f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-10T06:36:16.627 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:16 smithi139 podman[130048]: 2024-02-10 06:36:16.627133626 +0000 UTC m=+0.397247402 container init f7d93cf586b1e5c671501068e3568534f98bbda21f94b75063c26f05aa0312f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:36:16.920 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:16 smithi139 podman[130048]: 2024-02-10 06:36:16.630803696 +0000 UTC m=+0.400917473 container start f7d93cf586b1e5c671501068e3568534f98bbda21f94b75063c26f05aa0312f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131) 2024-02-10T06:36:16.920 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:16 smithi139 podman[130048]: 2024-02-10 06:36:16.718114335 +0000 UTC m=+0.488228113 container attach f7d93cf586b1e5c671501068e3568534f98bbda21f94b75063c26f05aa0312f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-10T06:36:17.204 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 10 06:36:16 smithi139 podman[130048]: 2024-02-10 06:36:16.920464336 +0000 UTC m=+0.690578104 container died f7d93cf586b1e5c671501068e3568534f98bbda21f94b75063c26f05aa0312f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-10T06:36:17.508 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.0.service' 2024-02-10T06:36:17.549 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:36:17.550 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-10T06:36:17.550 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-10T06:36:17.550 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.1 2024-02-10T06:36:17.836 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:17 smithi139 systemd[1]: Stopping Ceph osd.1 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:18.336 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:17 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[100318]: 2024-02-10T06:36:17.966+0000 7f91df8e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:18.336 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:17 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[100318]: 2024-02-10T06:36:17.966+0000 7f91df8e9700 -1 osd.1 161 *** Got signal Terminated *** 2024-02-10T06:36:18.336 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:17 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1[100318]: 2024-02-10T06:36:17.966+0000 7f91df8e9700 -1 osd.1 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:36:23.726 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:23 smithi139 podman[130152]: 2024-02-10 06:36:23.433663211 +0000 UTC m=+5.662856501 container died e33be38d0736264c6c736498e9011dc3294c44fbe353c5f8aaa34cf10b21f6f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True) 2024-02-10T06:36:24.058 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:23 smithi139 podman[130152]: 2024-02-10 06:36:23.799160633 +0000 UTC m=+6.028353973 container cleanup e33be38d0736264c6c736498e9011dc3294c44fbe353c5f8aaa34cf10b21f6f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-10T06:36:24.059 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:23 smithi139 bash[130152]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1 2024-02-10T06:36:24.337 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:24 smithi139 podman[130171]: 2024-02-10 06:36:24.058528549 +0000 UTC m=+0.624670521 container remove e33be38d0736264c6c736498e9011dc3294c44fbe353c5f8aaa34cf10b21f6f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, RELEASE=main-290212a, ceph=True, org.label-schema.license=GPLv2) 2024-02-10T06:36:24.731 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:24 smithi139 podman[130262]: 2024-02-10 06:36:24.417610699 +0000 UTC m=+0.113356738 container create bfc4af89d04afca10cefddd63b7e150d46b44e8c5338579d65f422a0ad9169c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-290212a) 2024-02-10T06:36:24.731 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:24 smithi139 podman[130262]: 2024-02-10 06:36:24.322313621 +0000 UTC m=+0.018059671 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:36:24.731 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:24 smithi139 podman[130262]: 2024-02-10 06:36:24.628506799 +0000 UTC m=+0.324252853 container init bfc4af89d04afca10cefddd63b7e150d46b44e8c5338579d65f422a0ad9169c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD) 2024-02-10T06:36:24.732 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:24 smithi139 podman[130262]: 2024-02-10 06:36:24.631950318 +0000 UTC m=+0.327696363 container start bfc4af89d04afca10cefddd63b7e150d46b44e8c5338579d65f422a0ad9169c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:36:24.732 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:24 smithi139 podman[130262]: 2024-02-10 06:36:24.728213095 +0000 UTC m=+0.423959141 container attach bfc4af89d04afca10cefddd63b7e150d46b44e8c5338579d65f422a0ad9169c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:36:25.087 INFO:journalctl@ceph.osd.1.smithi139.stdout:Feb 10 06:36:24 smithi139 podman[130262]: 2024-02-10 06:36:24.918716516 +0000 UTC m=+0.614462557 container died bfc4af89d04afca10cefddd63b7e150d46b44e8c5338579d65f422a0ad9169c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:36:25.554 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.1.service' 2024-02-10T06:36:25.596 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:36:25.596 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-10T06:36:25.597 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-10T06:36:25.597 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.2 2024-02-10T06:36:25.836 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:25 smithi139 systemd[1]: Stopping Ceph osd.2 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:26.336 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:25 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[106544]: 2024-02-10T06:36:25.916+0000 7f1695f49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:26.337 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:25 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[106544]: 2024-02-10T06:36:25.916+0000 7f1695f49700 -1 osd.2 161 *** Got signal Terminated *** 2024-02-10T06:36:26.337 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:25 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2[106544]: 2024-02-10T06:36:25.916+0000 7f1695f49700 -1 osd.2 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:36:31.627 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:31 smithi139 podman[130366]: 2024-02-10 06:36:31.363155674 +0000 UTC m=+5.548969763 container died 8d3bec1b004008633967f9d0bd957d16277598719975647837c3024b091c68e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-02-10T06:36:31.993 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:31 smithi139 podman[130366]: 2024-02-10 06:36:31.717249305 +0000 UTC m=+5.903063386 container cleanup 8d3bec1b004008633967f9d0bd957d16277598719975647837c3024b091c68e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-10T06:36:31.993 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:31 smithi139 bash[130366]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2 2024-02-10T06:36:32.331 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:31 smithi139 podman[130383]: 2024-02-10 06:36:31.993449832 +0000 UTC m=+0.630103910 container remove 8d3bec1b004008633967f9d0bd957d16277598719975647837c3024b091c68e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-10T06:36:32.583 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:32 smithi139 podman[130476]: 2024-02-10 06:36:32.233383544 +0000 UTC m=+0.017680976 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:36:32.583 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:32 smithi139 podman[130476]: 2024-02-10 06:36:32.352399417 +0000 UTC m=+0.136696834 container create 578fb2f0bdff23a1b39a7086a0461cbd0d47c53fb7b818fc0b8cf42537d9247d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2) 2024-02-10T06:36:32.584 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:32 smithi139 podman[130476]: 2024-02-10 06:36:32.580290377 +0000 UTC m=+0.364587798 container init 578fb2f0bdff23a1b39a7086a0461cbd0d47c53fb7b818fc0b8cf42537d9247d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:36:32.584 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:32 smithi139 podman[130476]: 2024-02-10 06:36:32.583347873 +0000 UTC m=+0.367645292 container start 578fb2f0bdff23a1b39a7086a0461cbd0d47c53fb7b818fc0b8cf42537d9247d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-10T06:36:32.836 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:32 smithi139 podman[130476]: 2024-02-10 06:36:32.697009245 +0000 UTC m=+0.481306664 container attach 578fb2f0bdff23a1b39a7086a0461cbd0d47c53fb7b818fc0b8cf42537d9247d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-10T06:36:33.158 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:32 smithi139 podman[130476]: 2024-02-10 06:36:32.874837466 +0000 UTC m=+0.659134881 container died 578fb2f0bdff23a1b39a7086a0461cbd0d47c53fb7b818fc0b8cf42537d9247d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:36:33.408 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:33 smithi139 podman[130503]: 2024-02-10 06:36:33.391717316 +0000 UTC m=+0.511849007 container remove 578fb2f0bdff23a1b39a7086a0461cbd0d47c53fb7b818fc0b8cf42537d9247d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-10T06:36:33.408 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:33 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-10T06:36:33.408 INFO:journalctl@ceph.osd.2.smithi139.stdout:Feb 10 06:36:33 smithi139 systemd[1]: Stopped Ceph osd.2 for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:36:33.411 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.2.service' 2024-02-10T06:36:33.453 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:36:33.453 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-10T06:36:33.453 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-10T06:36:33.453 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.3 2024-02-10T06:36:33.753 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:33 smithi139 systemd[1]: Stopping Ceph osd.3 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:33.753 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:36:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:36:33.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-02-10T06:36:33.753 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:36:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:36:33.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi150.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused" 2024-02-10T06:36:34.086 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[113189]: 2024-02-10T06:36:33.752+0000 7f05e15c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:34.086 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[113189]: 2024-02-10T06:36:33.752+0000 7f05e15c5700 -1 osd.3 161 *** Got signal Terminated *** 2024-02-10T06:36:34.086 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:33 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3[113189]: 2024-02-10T06:36:33.752+0000 7f05e15c5700 -1 osd.3 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:36:36.624 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:36 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:36.231+0000 7fccc2039700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:10.047475+0000 front 2024-02-10T06:36:10.047426+0000 (oldest deadline 2024-02-10T06:36:35.347331+0000) 2024-02-10T06:36:37.281 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:37.007+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:37.624 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:37 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:37.280+0000 7fccc2039700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:10.047475+0000 front 2024-02-10T06:36:10.047426+0000 (oldest deadline 2024-02-10T06:36:35.347331+0000) 2024-02-10T06:36:38.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:38.021+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:38.624 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:38 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:38.243+0000 7fccc2039700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:10.047475+0000 front 2024-02-10T06:36:10.047426+0000 (oldest deadline 2024-02-10T06:36:35.347331+0000) 2024-02-10T06:36:39.326 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:39.245+0000 7fccc2039700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:10.047475+0000 front 2024-02-10T06:36:10.047426+0000 (oldest deadline 2024-02-10T06:36:35.347331+0000) 2024-02-10T06:36:39.326 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:39.000+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:39.461 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:39 smithi139 podman[130583]: 2024-02-10 06:36:39.20228642 +0000 UTC m=+5.527760403 container died e1826d3dcabe858826a6f259fc6005b0ecc8ecc147a86ebee67a4592482087f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-02-10T06:36:39.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:39.533+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:39.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:39 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:39.325+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:39.836 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:39 smithi139 podman[130583]: 2024-02-10 06:36:39.560213514 +0000 UTC m=+5.885687495 container cleanup e1826d3dcabe858826a6f259fc6005b0ecc8ecc147a86ebee67a4592482087f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2) 2024-02-10T06:36:39.836 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:39 smithi139 bash[130583]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3 2024-02-10T06:36:40.178 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:39 smithi139 podman[130600]: 2024-02-10 06:36:39.857515212 +0000 UTC m=+0.651366939 container remove e1826d3dcabe858826a6f259fc6005b0ecc8ecc147a86ebee67a4592482087f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:36:40.291 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:40 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:40.016+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:40.432 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:40 smithi139 podman[130694]: 2024-02-10 06:36:40.080893931 +0000 UTC m=+0.017925322 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:36:40.432 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:40 smithi139 podman[130694]: 2024-02-10 06:36:40.187375208 +0000 UTC m=+0.124406599 container create 7a8007ec64857f2c0524bbb4ef525d0307b7a011613115e096e81ea77b7ead2a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-290212a, GIT_BRANCH=HEAD) 2024-02-10T06:36:40.432 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:40 smithi139 podman[130694]: 2024-02-10 06:36:40.431469692 +0000 UTC m=+0.368501074 container init 7a8007ec64857f2c0524bbb4ef525d0307b7a011613115e096e81ea77b7ead2a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-10T06:36:40.572 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:40 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:40.290+0000 7fccc2039700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:10.047475+0000 front 2024-02-10T06:36:10.047426+0000 (oldest deadline 2024-02-10T06:36:35.347331+0000) 2024-02-10T06:36:40.572 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:40 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:40.322+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:40.719 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:40 smithi139 podman[130694]: 2024-02-10 06:36:40.434828814 +0000 UTC m=+0.371860195 container start 7a8007ec64857f2c0524bbb4ef525d0307b7a011613115e096e81ea77b7ead2a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-290212a, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:36:40.720 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:40 smithi139 podman[130694]: 2024-02-10 06:36:40.506435838 +0000 UTC m=+0.443467222 container attach 7a8007ec64857f2c0524bbb4ef525d0307b7a011613115e096e81ea77b7ead2a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , ceph=True) 2024-02-10T06:36:40.874 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:40 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:40.571+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:41.001 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:40 smithi139 podman[130694]: 2024-02-10 06:36:40.719923497 +0000 UTC m=+0.656954878 container died 7a8007ec64857f2c0524bbb4ef525d0307b7a011613115e096e81ea77b7ead2a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T06:36:41.261 INFO:journalctl@ceph.osd.3.smithi139.stdout:Feb 10 06:36:41 smithi139 podman[130720]: 2024-02-10 06:36:41.242860424 +0000 UTC m=+0.517941212 container remove 7a8007ec64857f2c0524bbb4ef525d0307b7a011613115e096e81ea77b7ead2a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T06:36:41.266 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.3.service' 2024-02-10T06:36:41.289 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:41.013+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:41.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:36:41.307 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-10T06:36:41.308 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-10T06:36:41.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.4 2024-02-10T06:36:41.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:41.528+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:41.625 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:41.299+0000 7fccc2039700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:10.047475+0000 front 2024-02-10T06:36:10.047426+0000 (oldest deadline 2024-02-10T06:36:35.347331+0000) 2024-02-10T06:36:41.625 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:41 smithi150 systemd[1]: Stopping Ceph osd.4 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:41.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:41.289+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:41.626 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:36:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:36:41.450Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.139:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.139:8765: connect: connection refused" 2024-02-10T06:36:41.626 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:36:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:36:41.455Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.139:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.139:8765: connect: connection refused" 2024-02-10T06:36:41.626 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:36:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:36:41.455Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.139:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.139:8765: connect: connection refused" 2024-02-10T06:36:41.626 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:36:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:36:41.455Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.139:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.139:8765: connect: connection refused" 2024-02-10T06:36:42.022 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:41.642+0000 7fccc9a4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:42.022 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:41.642+0000 7fccc9a4d700 -1 osd.4 161 *** Got signal Terminated *** 2024-02-10T06:36:42.022 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:41 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:41.642+0000 7fccc9a4d700 -1 osd.4 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:36:42.297 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:42 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:42.249+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:42.297 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:42 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:42.021+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:42.624 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:42 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:42.296+0000 7fccc2039700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:10.047475+0000 front 2024-02-10T06:36:10.047426+0000 (oldest deadline 2024-02-10T06:36:35.347331+0000) 2024-02-10T06:36:42.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:42 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:42.486+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:43.314 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:43.043+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:43.315 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:43.234+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:43.624 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:43.313+0000 7fccc2039700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:10.047475+0000 front 2024-02-10T06:36:10.047426+0000 (oldest deadline 2024-02-10T06:36:35.347331+0000) 2024-02-10T06:36:43.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:43 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:43.448+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:43.837 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:36:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:36:43.552Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi150.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused" 2024-02-10T06:36:43.837 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:36:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:36:43.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-02-10T06:36:43.837 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:36:43 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:36:43.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi150.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused" 2024-02-10T06:36:44.374 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:44 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:44.308+0000 7fccc2039700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:10.047475+0000 front 2024-02-10T06:36:10.047426+0000 (oldest deadline 2024-02-10T06:36:35.347331+0000) 2024-02-10T06:36:44.375 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:44 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:44.221+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:44.375 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:44 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:44.059+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:44.874 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:44 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:44.491+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:45.355 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:45.012+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:45.355 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:45.199+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:45.624 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:45.354+0000 7fccc2039700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:10.047475+0000 front 2024-02-10T06:36:10.047426+0000 (oldest deadline 2024-02-10T06:36:35.347331+0000) 2024-02-10T06:36:45.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:45.459+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:46.374 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:46.218+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:46.375 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:45.996+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:46.375 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:45 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:45.996+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:46.875 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4[78822]: 2024-02-10T06:36:46.394+0000 7fccc2039700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:10.047475+0000 front 2024-02-10T06:36:10.047426+0000 (oldest deadline 2024-02-10T06:36:35.347331+0000) 2024-02-10T06:36:46.875 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:46.427+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:47.235 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:46 smithi150 podman[102165]: 2024-02-10 06:36:46.988101343 +0000 UTC m=+5.484248607 container died f61046f26c977333ff2e22978034a16770c2c24c061346455c3467aa2b137271 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-10T06:36:47.236 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:46.947+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:47.236 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:46 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:46.947+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:47.607 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:47.406+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:47.607 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:47 smithi150 podman[102165]: 2024-02-10 06:36:47.422442508 +0000 UTC m=+5.918589758 container cleanup f61046f26c977333ff2e22978034a16770c2c24c061346455c3467aa2b137271 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-10T06:36:47.607 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:47 smithi150 bash[102165]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4 2024-02-10T06:36:47.608 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:47.234+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:47.868 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:47 smithi150 podman[102183]: 2024-02-10 06:36:47.607151792 +0000 UTC m=+0.622152420 container remove f61046f26c977333ff2e22978034a16770c2c24c061346455c3467aa2b137271 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, ceph=True, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-10T06:36:48.124 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:47.905+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:48.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:47 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:47.905+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:48.125 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:48 smithi150 podman[102273]: 2024-02-10 06:36:47.935548723 +0000 UTC m=+0.018340961 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:36:48.125 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:48 smithi150 podman[102273]: 2024-02-10 06:36:48.03288929 +0000 UTC m=+0.115681521 container create a08079328a7972f5a1c5e4cff52f39f9fc39534e256ad90ad2c39b8c0903926b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True) 2024-02-10T06:36:48.411 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:48.212+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:48.412 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:48.212+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.883352+0000 front 2024-02-10T06:36:22.883212+0000 (oldest deadline 2024-02-10T06:36:47.583234+0000) 2024-02-10T06:36:48.412 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:48 smithi150 podman[102273]: 2024-02-10 06:36:48.235588919 +0000 UTC m=+0.318381148 container init a08079328a7972f5a1c5e4cff52f39f9fc39534e256ad90ad2c39b8c0903926b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-10T06:36:48.412 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:48 smithi150 podman[102273]: 2024-02-10 06:36:48.238678685 +0000 UTC m=+0.321470909 container start a08079328a7972f5a1c5e4cff52f39f9fc39534e256ad90ad2c39b8c0903926b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:36:48.412 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:48 smithi150 podman[102273]: 2024-02-10 06:36:48.310707131 +0000 UTC m=+0.393499365 container attach a08079328a7972f5a1c5e4cff52f39f9fc39534e256ad90ad2c39b8c0903926b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:36:48.822 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 10 06:36:48 smithi150 podman[102273]: 2024-02-10 06:36:48.531552379 +0000 UTC m=+0.614344609 container died a08079328a7972f5a1c5e4cff52f39f9fc39534e256ad90ad2c39b8c0903926b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a) 2024-02-10T06:36:48.822 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:48.410+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:48.823 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:48.410+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:49.124 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:48.953+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:49.124 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:48 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:48.953+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:49.182 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.4.service' 2024-02-10T06:36:49.226 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:36:49.227 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-10T06:36:49.227 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-10T06:36:49.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.5 2024-02-10T06:36:49.418 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:49.363+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:49.419 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:49.363+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:49.419 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:49.223+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:49.419 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:49.223+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.883352+0000 front 2024-02-10T06:36:22.883212+0000 (oldest deadline 2024-02-10T06:36:47.583234+0000) 2024-02-10T06:36:49.419 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:49 smithi150 systemd[1]: Stopping Ceph osd.5 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:49.874 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:49.592+0000 7f3c4a367700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:49.875 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:49.592+0000 7f3c4a367700 -1 osd.5 161 *** Got signal Terminated *** 2024-02-10T06:36:49.875 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:49.592+0000 7f3c4a367700 -1 osd.5 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:36:50.200 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:49.920+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:50.201 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:49 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:49.920+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:50.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:50.199+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:50.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:50.199+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.883352+0000 front 2024-02-10T06:36:22.883212+0000 (oldest deadline 2024-02-10T06:36:47.583234+0000) 2024-02-10T06:36:50.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:50.372+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:50.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:50.372+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:51.224 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:50.920+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:51.224 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:50 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:50.920+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:51.624 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:51.342+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:51.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:51.342+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:51.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:51.223+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:51.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:51.223+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.883352+0000 front 2024-02-10T06:36:22.883212+0000 (oldest deadline 2024-02-10T06:36:47.583234+0000) 2024-02-10T06:36:52.253 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:51.875+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:52.253 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:51 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:51.875+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:52.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:52.252+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:52.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:52.252+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.883352+0000 front 2024-02-10T06:36:22.883212+0000 (oldest deadline 2024-02-10T06:36:47.583234+0000) 2024-02-10T06:36:52.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:52.301+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:52.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:52.301+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:52.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:52.301+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:36:53.265 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:52.880+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:53.265 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:52.880+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:53.265 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:52 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:52.880+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:30.849320+0000 front 2024-02-10T06:36:30.849266+0000 (oldest deadline 2024-02-10T06:36:51.949252+0000) 2024-02-10T06:36:53.624 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:53 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:53.271+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:53.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:53 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:53.271+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.883352+0000 front 2024-02-10T06:36:22.883212+0000 (oldest deadline 2024-02-10T06:36:47.583234+0000) 2024-02-10T06:36:53.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:53 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:53.264+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:53.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:53 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:53.264+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:53.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:53 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:53.264+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:36:53.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:36:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:36:53.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi150.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-02-10T06:36:53.837 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:36:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:36:53.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-02-10T06:36:53.837 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:36:53 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:36:53.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi150.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused" 2024-02-10T06:36:54.270 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:53 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:53.920+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:54.270 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:53 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:53.920+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:54.270 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:53 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:53.920+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:30.849320+0000 front 2024-02-10T06:36:30.849266+0000 (oldest deadline 2024-02-10T06:36:51.949252+0000) 2024-02-10T06:36:54.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:54 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:54.269+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.081320+0000 front 2024-02-10T06:36:13.081377+0000 (oldest deadline 2024-02-10T06:36:38.981169+0000) 2024-02-10T06:36:54.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:54 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:54.269+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.883352+0000 front 2024-02-10T06:36:22.883212+0000 (oldest deadline 2024-02-10T06:36:47.583234+0000) 2024-02-10T06:36:54.625 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:54 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5[83366]: 2024-02-10T06:36:54.269+0000 7f3c4695b700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:27.583953+0000 front 2024-02-10T06:36:27.583909+0000 (oldest deadline 2024-02-10T06:36:53.483870+0000) 2024-02-10T06:36:54.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:54 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:54.284+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:54.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:54 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:54.284+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:54.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:54 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:54.284+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:36:55.203 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:54 smithi150 podman[102378]: 2024-02-10 06:36:54.934344424 +0000 UTC m=+5.478561977 container died 24270c0f3351c4889b6b29bf02d3ae59dd0352facb3bd1178d7c62e617fe2f2d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-10T06:36:55.204 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:54 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:54.899+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:55.204 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:54 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:54.899+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:55.204 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:54 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:54.899+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:30.849320+0000 front 2024-02-10T06:36:30.849266+0000 (oldest deadline 2024-02-10T06:36:51.949252+0000) 2024-02-10T06:36:55.554 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:55 smithi150 podman[102378]: 2024-02-10 06:36:55.368754233 +0000 UTC m=+5.912971819 container cleanup 24270c0f3351c4889b6b29bf02d3ae59dd0352facb3bd1178d7c62e617fe2f2d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-10T06:36:55.554 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:55 smithi150 bash[102378]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5 2024-02-10T06:36:55.554 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:55 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:55.240+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:55.554 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:55 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:55.240+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:55.554 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:55 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:55.240+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:36:55.872 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:55 smithi150 podman[102407]: 2024-02-10 06:36:55.5534908 +0000 UTC m=+0.613469452 container remove 24270c0f3351c4889b6b29bf02d3ae59dd0352facb3bd1178d7c62e617fe2f2d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-10T06:36:56.124 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:55 smithi150 podman[102497]: 2024-02-10 06:36:55.883868326 +0000 UTC m=+0.018009218 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:36:56.125 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:55 smithi150 podman[102497]: 2024-02-10 06:36:55.987677383 +0000 UTC m=+0.121818309 container create 95104a036379e5546c39cf50c14ad5ec1fed6f4c8d270e4ba03fa6fe1c9644b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-10T06:36:56.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:55 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:55.871+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:56.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:55 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:55.871+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:56.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:55 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:55.871+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:30.849320+0000 front 2024-02-10T06:36:30.849266+0000 (oldest deadline 2024-02-10T06:36:51.949252+0000) 2024-02-10T06:36:56.511 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:56.202+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:56.511 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:56.202+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:56.511 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:56.202+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:36:56.512 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:56 smithi150 podman[102497]: 2024-02-10 06:36:56.215431069 +0000 UTC m=+0.349571962 container init 95104a036379e5546c39cf50c14ad5ec1fed6f4c8d270e4ba03fa6fe1c9644b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-10T06:36:56.512 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:56 smithi150 podman[102497]: 2024-02-10 06:36:56.218656724 +0000 UTC m=+0.352797608 container start 95104a036379e5546c39cf50c14ad5ec1fed6f4c8d270e4ba03fa6fe1c9644b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-02-10T06:36:56.512 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:56 smithi150 podman[102497]: 2024-02-10 06:36:56.290196897 +0000 UTC m=+0.424337786 container attach 95104a036379e5546c39cf50c14ad5ec1fed6f4c8d270e4ba03fa6fe1c9644b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:36:56.793 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 10 06:36:56 smithi150 podman[102497]: 2024-02-10 06:36:56.5111358 +0000 UTC m=+0.645276701 container died 95104a036379e5546c39cf50c14ad5ec1fed6f4c8d270e4ba03fa6fe1c9644b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-10T06:36:57.124 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:56.848+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:57.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:56.848+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:57.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:56 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:56.848+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:30.849320+0000 front 2024-02-10T06:36:30.849266+0000 (oldest deadline 2024-02-10T06:36:51.949252+0000) 2024-02-10T06:36:57.171 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.5.service' 2024-02-10T06:36:57.213 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:36:57.213 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-10T06:36:57.213 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-10T06:36:57.213 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.6 2024-02-10T06:36:57.522 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:57 smithi150 systemd[1]: Stopping Ceph osd.6 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:36:57.523 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:57.237+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:57.523 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:57.237+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:57.523 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:57.237+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:36:57.803 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:57.521+0000 7fd60b369700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:57.803 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:57.521+0000 7fd60b369700 -1 osd.6 161 *** Got signal Terminated *** 2024-02-10T06:36:57.803 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:57.521+0000 7fd60b369700 -1 osd.6 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:36:57.803 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:57.802+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:57.803 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:57.802+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:57.803 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:57 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:57.802+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:30.849320+0000 front 2024-02-10T06:36:30.849266+0000 (oldest deadline 2024-02-10T06:36:51.949252+0000) 2024-02-10T06:36:58.624 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:58.207+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:58.624 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:58.207+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:58.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:58.207+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:36:59.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:58.851+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:36:59.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:58.851+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:36:59.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:58 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:58.851+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:30.849320+0000 front 2024-02-10T06:36:30.849266+0000 (oldest deadline 2024-02-10T06:36:51.949252+0000) 2024-02-10T06:36:59.624 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:59.236+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:36:59.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:59.236+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:36:59.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:36:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:36:59.236+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:00.124 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:59.859+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:37:00.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:59.859+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:37:00.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:36:59 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:36:59.859+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:30.849320+0000 front 2024-02-10T06:36:30.849266+0000 (oldest deadline 2024-02-10T06:36:51.949252+0000) 2024-02-10T06:37:00.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:00.198+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:37:00.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:00.198+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:37:00.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:00.198+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:00.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:00.198+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6830 osd.3 since back 2024-02-10T06:36:36.438088+0000 front 2024-02-10T06:36:36.438044+0000 (oldest deadline 2024-02-10T06:36:59.937989+0000) 2024-02-10T06:37:01.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:37:00.871+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:37:01.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:37:00.871+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:37:01.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:00 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:37:00.871+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:30.849320+0000 front 2024-02-10T06:36:30.849266+0000 (oldest deadline 2024-02-10T06:36:51.949252+0000) 2024-02-10T06:37:01.624 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:01.170+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:37:01.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:01.170+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:37:01.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:01.170+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:01.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:01.170+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6830 osd.3 since back 2024-02-10T06:36:36.438088+0000 front 2024-02-10T06:36:36.438044+0000 (oldest deadline 2024-02-10T06:36:59.937989+0000) 2024-02-10T06:37:02.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:37:01.866+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:11.646387+0000 front 2024-02-10T06:36:11.646469+0000 (oldest deadline 2024-02-10T06:36:36.946573+0000) 2024-02-10T06:37:02.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:37:01.866+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:19.247569+0000 front 2024-02-10T06:36:19.247496+0000 (oldest deadline 2024-02-10T06:36:45.147474+0000) 2024-02-10T06:37:02.125 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:01 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6[88327]: 2024-02-10T06:37:01.866+0000 7fd60795d700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:30.849320+0000 front 2024-02-10T06:36:30.849266+0000 (oldest deadline 2024-02-10T06:36:51.949252+0000) 2024-02-10T06:37:02.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:02.136+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:37:02.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:02.136+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:37:02.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:02.136+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:02.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:02.136+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6830 osd.3 since back 2024-02-10T06:36:36.438088+0000 front 2024-02-10T06:36:36.438044+0000 (oldest deadline 2024-02-10T06:36:59.937989+0000) 2024-02-10T06:37:02.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:02 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:02.136+0000 7fa1267fd700 -1 osd.7 161 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.44558.0:42 2.8 2:166d84f2:::realms.5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028.control:head [watch ping cookie 94554138713984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161) 2024-02-10T06:37:03.124 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:02 smithi150 podman[102602]: 2024-02-10 06:37:02.834799457 +0000 UTC m=+5.397297890 container died 6136d2cdfcc2355b78394c3fd79c7e5f4ac9b235cd60c00f6b163aab06e67b62 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-10T06:37:03.389 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:03 smithi150 podman[102602]: 2024-02-10 06:37:03.154869887 +0000 UTC m=+5.717368313 container cleanup 6136d2cdfcc2355b78394c3fd79c7e5f4ac9b235cd60c00f6b163aab06e67b62 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-10T06:37:03.389 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:03 smithi150 bash[102602]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6 2024-02-10T06:37:03.390 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:03 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:03.127+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:37:03.390 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:03 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:03.127+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:37:03.390 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:03 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:03.127+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:03.390 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:03 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:03.127+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6830 osd.3 since back 2024-02-10T06:36:36.438088+0000 front 2024-02-10T06:36:36.438044+0000 (oldest deadline 2024-02-10T06:36:59.937989+0000) 2024-02-10T06:37:03.390 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:03 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:03.127+0000 7fa1267fd700 -1 osd.7 161 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.44558.0:42 2.8 2:166d84f2:::realms.5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028.control:head [watch ping cookie 94554138713984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161) 2024-02-10T06:37:03.823 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:03 smithi150 podman[102619]: 2024-02-10 06:37:03.389445253 +0000 UTC m=+0.547335599 container remove 6136d2cdfcc2355b78394c3fd79c7e5f4ac9b235cd60c00f6b163aab06e67b62 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:37:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:37:03.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi150.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-02-10T06:37:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:37:03.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-02-10T06:37:03.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:03 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:37:03.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi150.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused" 2024-02-10T06:37:04.083 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:03 smithi150 podman[102708]: 2024-02-10 06:37:03.823724814 +0000 UTC m=+0.108645264 container create af4bb9b3978193f0efcbc4d9f2d0fcaf0745eb90ae8a4795c05e86290568c03b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:37:04.083 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:03 smithi150 podman[102708]: 2024-02-10 06:37:03.733792038 +0000 UTC m=+0.018712501 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:37:04.083 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:04 smithi150 podman[102708]: 2024-02-10 06:37:04.051357124 +0000 UTC m=+0.336277578 container init af4bb9b3978193f0efcbc4d9f2d0fcaf0745eb90ae8a4795c05e86290568c03b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:37:04.084 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:04 smithi150 podman[102708]: 2024-02-10 06:37:04.055246296 +0000 UTC m=+0.340166753 container start af4bb9b3978193f0efcbc4d9f2d0fcaf0745eb90ae8a4795c05e86290568c03b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-10T06:37:04.341 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:04 smithi150 podman[102708]: 2024-02-10 06:37:04.126551203 +0000 UTC m=+0.411471660 container attach af4bb9b3978193f0efcbc4d9f2d0fcaf0745eb90ae8a4795c05e86290568c03b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131) 2024-02-10T06:37:04.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:04.081+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:37:04.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:04.081+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:37:04.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:04.081+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:04.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:04.081+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6830 osd.3 since back 2024-02-10T06:36:36.438088+0000 front 2024-02-10T06:36:36.438044+0000 (oldest deadline 2024-02-10T06:36:59.937989+0000) 2024-02-10T06:37:04.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:04 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:04.082+0000 7fa1267fd700 -1 osd.7 161 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.44558.0:42 2.8 2:166d84f2:::realms.5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028.control:head [watch ping cookie 94554138713984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161) 2024-02-10T06:37:04.621 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 10 06:37:04 smithi150 podman[102708]: 2024-02-10 06:37:04.341739938 +0000 UTC m=+0.626660401 container died af4bb9b3978193f0efcbc4d9f2d0fcaf0745eb90ae8a4795c05e86290568c03b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-10T06:37:04.987 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.6.service' 2024-02-10T06:37:05.028 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:37:05.029 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-10T06:37:05.029 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-10T06:37:05.029 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.7 2024-02-10T06:37:05.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:05.107+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:37:05.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:05.107+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:37:05.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:05.107+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:05.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:05.107+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6830 osd.3 since back 2024-02-10T06:36:36.438088+0000 front 2024-02-10T06:36:36.438044+0000 (oldest deadline 2024-02-10T06:36:59.937989+0000) 2024-02-10T06:37:05.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:05.107+0000 7fa1267fd700 -1 osd.7 161 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.44558.0:42 2.8 2:166d84f2:::realms.5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028.control:head [watch ping cookie 94554138713984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161) 2024-02-10T06:37:05.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:05 smithi150 systemd[1]: Stopping Ceph osd.7 for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:37:05.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:05.340+0000 7fa12e211700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:37:05.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:05.340+0000 7fa12e211700 -1 osd.7 161 *** Got signal Terminated *** 2024-02-10T06:37:05.342 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:05 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:05.340+0000 7fa12e211700 -1 osd.7 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:37:06.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:06.081+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:37:06.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:06.081+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:37:06.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:06.082+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:06.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:06.082+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6830 osd.3 since back 2024-02-10T06:36:36.438088+0000 front 2024-02-10T06:36:36.438044+0000 (oldest deadline 2024-02-10T06:36:59.937989+0000) 2024-02-10T06:37:06.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:06 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:06.082+0000 7fa1267fd700 -1 osd.7 161 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.44558.0:42 2.8 2:166d84f2:::realms.5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028.control:head [watch ping cookie 94554138713984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161) 2024-02-10T06:37:07.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:07.085+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:37:07.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:07.085+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:37:07.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:07.085+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:07.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:07.085+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6830 osd.3 since back 2024-02-10T06:36:36.438088+0000 front 2024-02-10T06:36:36.438044+0000 (oldest deadline 2024-02-10T06:36:59.937989+0000) 2024-02-10T06:37:07.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:07 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:07.085+0000 7fa1267fd700 -1 osd.7 161 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.44558.0:42 2.8 2:166d84f2:::realms.5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028.control:head [watch ping cookie 94554138713984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161) 2024-02-10T06:37:08.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:08.095+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:37:08.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:08.095+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:37:08.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:08.095+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:08.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:08.095+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6830 osd.3 since back 2024-02-10T06:36:36.438088+0000 front 2024-02-10T06:36:36.438044+0000 (oldest deadline 2024-02-10T06:36:59.937989+0000) 2024-02-10T06:37:08.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:08 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:08.095+0000 7fa1267fd700 -1 osd.7 161 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.44558.0:42 2.8 2:166d84f2:::realms.5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028.control:head [watch ping cookie 94554138713984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161) 2024-02-10T06:37:09.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:09.047+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:37:09.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:09.047+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:37:09.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:09.047+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:09.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:09.047+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6830 osd.3 since back 2024-02-10T06:36:36.438088+0000 front 2024-02-10T06:36:36.438044+0000 (oldest deadline 2024-02-10T06:36:59.937989+0000) 2024-02-10T06:37:09.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:09.047+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.150:6804 osd.4 since back 2024-02-10T06:36:45.739129+0000 front 2024-02-10T06:36:45.739199+0000 (oldest deadline 2024-02-10T06:37:08.639108+0000) 2024-02-10T06:37:09.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:09 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:09.047+0000 7fa1267fd700 -1 osd.7 161 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.44558.0:42 2.8 2:166d84f2:::realms.5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028.control:head [watch ping cookie 94554138713984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161) 2024-02-10T06:37:10.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:10.065+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6806 osd.0 since back 2024-02-10T06:36:13.535634+0000 front 2024-02-10T06:36:13.535755+0000 (oldest deadline 2024-02-10T06:36:39.435670+0000) 2024-02-10T06:37:10.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:10.065+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6814 osd.1 since back 2024-02-10T06:36:22.936586+0000 front 2024-02-10T06:36:22.936616+0000 (oldest deadline 2024-02-10T06:36:48.236560+0000) 2024-02-10T06:37:10.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:10.065+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6822 osd.2 since back 2024-02-10T06:36:28.237131+0000 front 2024-02-10T06:36:28.237203+0000 (oldest deadline 2024-02-10T06:36:51.737107+0000) 2024-02-10T06:37:10.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:10.065+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.139:6830 osd.3 since back 2024-02-10T06:36:36.438088+0000 front 2024-02-10T06:36:36.438044+0000 (oldest deadline 2024-02-10T06:36:59.937989+0000) 2024-02-10T06:37:10.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:10.065+0000 7fa1267fd700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.150:6804 osd.4 since back 2024-02-10T06:36:45.739129+0000 front 2024-02-10T06:36:45.739199+0000 (oldest deadline 2024-02-10T06:37:08.639108+0000) 2024-02-10T06:37:10.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:10 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7[93319]: 2024-02-10T06:37:10.065+0000 7fa1267fd700 -1 osd.7 161 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.44558.0:42 2.8 2:166d84f2:::realms.5c1a1cb1-7c3d-4bef-a0c7-9c3c8e737028.control:head [watch ping cookie 94554138713984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161) 2024-02-10T06:37:11.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:10 smithi150 podman[102813]: 2024-02-10 06:37:10.832595314 +0000 UTC m=+5.582484885 container died 7818cf0fd8b527d604c9416794bd669711162dac69de53a158cc6d621476d9be (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-10T06:37:11.624 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:11 smithi150 podman[102813]: 2024-02-10 06:37:11.216251403 +0000 UTC m=+5.966140983 container cleanup 7818cf0fd8b527d604c9416794bd669711162dac69de53a158cc6d621476d9be (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0) 2024-02-10T06:37:11.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:11 smithi150 bash[102813]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7 2024-02-10T06:37:11.625 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:11 smithi150 podman[102841]: 2024-02-10 06:37:11.417980934 +0000 UTC m=+0.586532690 container remove 7818cf0fd8b527d604c9416794bd669711162dac69de53a158cc6d621476d9be (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-10T06:37:11.945 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:11 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:37:11.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-02-10T06:37:11.946 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:11 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:37:11.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi150.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused" 2024-02-10T06:37:11.996 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:11 smithi150 podman[102938]: 2024-02-10 06:37:11.710264825 +0000 UTC m=+0.105567223 container create b2e551a24abb110ed9880408a273dd71a362d3ad9bca7dcd5426ad80753d7498 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:37:11.996 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:11 smithi150 podman[102938]: 2024-02-10 06:37:11.623910244 +0000 UTC m=+0.019212654 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:37:11.996 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:11 smithi150 podman[102938]: 2024-02-10 06:37:11.912823891 +0000 UTC m=+0.308126290 container init b2e551a24abb110ed9880408a273dd71a362d3ad9bca7dcd5426ad80753d7498 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2) 2024-02-10T06:37:11.996 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:11 smithi150 podman[102938]: 2024-02-10 06:37:11.915873033 +0000 UTC m=+0.311175433 container start b2e551a24abb110ed9880408a273dd71a362d3ad9bca7dcd5426ad80753d7498 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-10T06:37:11.996 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:11 smithi150 podman[102938]: 2024-02-10 06:37:11.996078527 +0000 UTC m=+0.391380938 container attach b2e551a24abb110ed9880408a273dd71a362d3ad9bca7dcd5426ad80753d7498 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:37:12.374 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:12 smithi150 podman[102938]: 2024-02-10 06:37:12.196280076 +0000 UTC m=+0.591582474 container died b2e551a24abb110ed9880408a273dd71a362d3ad9bca7dcd5426ad80753d7498 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:37:12.755 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 10 06:37:12 smithi150 podman[102963]: 2024-02-10 06:37:12.732723082 +0000 UTC m=+0.526698344 container remove b2e551a24abb110ed9880408a273dd71a362d3ad9bca7dcd5426ad80753d7498 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T06:37:12.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@osd.7.service' 2024-02-10T06:37:12.802 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:37:12.802 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-10T06:37:12.802 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-10T06:37:12.802 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a 2024-02-10T06:37:13.124 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:12 smithi150 systemd[1]: Stopping Ceph prometheus.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:37:13.428 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.169Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:37:13.429 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.169Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-10T06:37:13.429 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.169Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-10T06:37:13.429 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.169Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T06:37:13.429 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.169Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-10T06:37:13.429 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.169Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-10T06:37:13.429 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.169Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T06:37:13.429 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.169Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-10T06:37:13.429 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.169Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-10T06:37:13.429 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.170Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T06:37:13.429 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.170Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-10T06:37:13.430 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a[64328]: ts=2024-02-10T06:37:13.170Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-10T06:37:13.430 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 podman[103044]: 2024-02-10 06:37:13.179951838 +0000 UTC m=+0.159663209 container died 8836434edb6082a0fcd740d5fee9682f0f9bcd824cd859babfb15efd7789ec5e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:37:13.795 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 podman[103044]: 2024-02-10 06:37:13.527427312 +0000 UTC m=+0.507138673 container cleanup 8836434edb6082a0fcd740d5fee9682f0f9bcd824cd859babfb15efd7789ec5e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:37:13.795 INFO:journalctl@ceph.prometheus.a.smithi150.stdout:Feb 10 06:37:13 smithi150 bash[103044]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T06:37:13.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:37:13.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi150.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused" 2024-02-10T06:37:13.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:37:13.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-02-10T06:37:13.837 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:13 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:37:13.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi150.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.150:8443: connect: connection refused" 2024-02-10T06:37:13.935 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@prometheus.a.service' 2024-02-10T06:37:13.970 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:37:13.971 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-10T06:37:13.971 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T06:37:15.837 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:37:15 smithi139 systemd[1]: Stopping Ceph node-exporter.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:37:15.837 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:37:15 smithi139 podman[131099]: 2024-02-10 06:37:15.728043232 +0000 UTC m=+0.099655748 container died 3c4db3eedf5e9da4475f81aa92423573d6f198209060ff58cc0248b1fbfa9640 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:37:16.228 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:37:16 smithi139 podman[131099]: 2024-02-10 06:37:16.050153126 +0000 UTC m=+0.421765642 container cleanup 3c4db3eedf5e9da4475f81aa92423573d6f198209060ff58cc0248b1fbfa9640 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:37:16.228 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:37:16 smithi139 bash[131099]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a 2024-02-10T06:37:16.228 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:37:16 smithi139 podman[131113]: 2024-02-10 06:37:16.226141712 +0000 UTC m=+0.502516789 container remove 3c4db3eedf5e9da4475f81aa92423573d6f198209060ff58cc0248b1fbfa9640 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:37:16.229 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:37:16 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:37:16.490 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:37:16 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T06:37:16.490 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:37:16 smithi139 systemd[1]: Stopped Ceph node-exporter.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:37:16.490 INFO:journalctl@ceph.node-exporter.a.smithi139.stdout:Feb 10 06:37:16 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Consumed 3.219s CPU time. 2024-02-10T06:37:16.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:16 smithi139 systemd[1]: Stopping Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:37:17.246 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:16 smithi139 ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80093]: ts=2024-02-10T06:37:16.920Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:37:17.246 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:16 smithi139 podman[131219]: 2024-02-10 06:37:16.971795732 +0000 UTC m=+0.174260922 container died 2b6a81902a4c8ccf4e1af789dac614e788d9656f4b3b3356f22c4541ed8b80e0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:37:17.580 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:17 smithi139 podman[131219]: 2024-02-10 06:37:17.345785232 +0000 UTC m=+0.548250432 container cleanup 2b6a81902a4c8ccf4e1af789dac614e788d9656f4b3b3356f22c4541ed8b80e0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:37:17.580 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:17 smithi139 bash[131219]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T06:37:17.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:17 smithi139 podman[131233]: 2024-02-10 06:37:17.580335616 +0000 UTC m=+0.613651071 container remove 2b6a81902a4c8ccf4e1af789dac614e788d9656f4b3b3356f22c4541ed8b80e0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:37:17.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:17 smithi139 podman[131233]: 2024-02-10 06:37:17.639262843 +0000 UTC m=+0.672578315 volume remove b00620828cc3d4f2d7de533b8940b2e78eeda555a0d9679e0c54feb57a9cccfe 2024-02-10T06:37:17.836 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:17 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T06:37:17.837 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:17 smithi139 systemd[1]: Stopped Ceph alertmanager.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:37:17.837 INFO:journalctl@ceph.alertmanager.a.smithi139.stdout:Feb 10 06:37:17 smithi139 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Consumed 1.615s CPU time. 2024-02-10T06:37:40.892 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T06:37:42.625 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:37:42 smithi150 systemd[1]: Stopping Ceph node-exporter.b for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:37:42.625 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:37:42 smithi150 podman[103442]: 2024-02-10 06:37:42.582185631 +0000 UTC m=+0.104143044 container died b250d2d9c43c27d6f36fb735b61baee03501f9569f62564506e3e0f136ebdac4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:37:43.081 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:37:42 smithi150 podman[103442]: 2024-02-10 06:37:42.904875326 +0000 UTC m=+0.426832733 container cleanup b250d2d9c43c27d6f36fb735b61baee03501f9569f62564506e3e0f136ebdac4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:37:43.081 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:37:42 smithi150 bash[103442]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b 2024-02-10T06:37:43.374 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:37:43 smithi150 podman[103456]: 2024-02-10 06:37:43.08095324 +0000 UTC m=+0.501894839 container remove b250d2d9c43c27d6f36fb735b61baee03501f9569f62564506e3e0f136ebdac4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:37:43.375 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:37:43 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:37:43.375 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:37:43 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T06:37:43.375 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:37:43 smithi150 systemd[1]: Stopped Ceph node-exporter.b for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:37:43.375 INFO:journalctl@ceph.node-exporter.b.smithi150.stdout:Feb 10 06:37:43 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Consumed 3.134s CPU time. 2024-02-10T06:37:43.736 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:37:43 smithi150 systemd[1]: Stopping Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:37:43.993 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:37:43 smithi150 podman[103566]: 2024-02-10 06:37:43.736123061 +0000 UTC m=+0.118923673 container died dae5565f9ce7d389d08ca511fd0f40ac271dab56d3089df2e6da8c5367e4b8b3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, version=8.5, io.buildah.version=1.32.0, vcs-type=git, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., name=ubi8, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, distribution-scope=public, maintainer=Paul Cuzner ) 2024-02-10T06:37:44.245 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:37:44 smithi150 podman[103566]: 2024-02-10 06:37:44.066915502 +0000 UTC m=+0.449716129 container cleanup dae5565f9ce7d389d08ca511fd0f40ac271dab56d3089df2e6da8c5367e4b8b3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, maintainer=Paul Cuzner , release=239.1651231664, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, version=8.5, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container) 2024-02-10T06:37:44.245 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:37:44 smithi150 bash[103566]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a 2024-02-10T06:37:44.245 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:37:44 smithi150 podman[103586]: 2024-02-10 06:37:44.243106488 +0000 UTC m=+0.504647565 container remove dae5565f9ce7d389d08ca511fd0f40ac271dab56d3089df2e6da8c5367e4b8b3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ce44e742-c7d8-11ee-95b8-87774f69a715-grafana-a, description=Ceph Grafana Container, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, com.redhat.component=ubi8-container, vcs-type=git, release=239.1651231664, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, architecture=x86_64, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-02-10T06:37:44.245 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:37:44 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:37:44.874 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:37:44 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T06:37:44.875 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:37:44 smithi150 systemd[1]: Stopped Ceph grafana.a for ce44e742-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:37:44.875 INFO:journalctl@ceph.grafana.a.smithi150.stdout:Feb 10 06:37:44 smithi150 systemd[1]: ceph-ce44e742-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Consumed 9.583s CPU time. 2024-02-10T06:37:56.188 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:37:56.224 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:37:56.254 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T06:37:56.257 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103/remote/smithi139/crash 2024-02-10T06:37:56.257 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/crash -- . 2024-02-10T06:37:56.313 INFO:teuthology.orchestra.run.smithi139.stderr:tar: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T06:37:56.313 INFO:teuthology.orchestra.run.smithi139.stderr:tar: Error is not recoverable: exiting now 2024-02-10T06:37:56.315 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103/remote/smithi150/crash 2024-02-10T06:37:56.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/crash -- . 2024-02-10T06:37:56.351 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T06:37:56.352 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2024-02-10T06:37:56.353 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T06:37:56.353 DEBUG:teuthology.orchestra.run.smithi139:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T06:37:56.385 INFO:teuthology.orchestra.run.smithi139.stdout:2024-02-10T05:56:17.726978+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-10T06:37:56.385 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T06:37:56.385 DEBUG:teuthology.orchestra.run.smithi139:> sudo egrep '\[SEC\]' /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T06:37:56.456 DEBUG:teuthology.orchestra.run.smithi139:> sudo egrep '\[ERR\]' /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T06:37:56.526 DEBUG:teuthology.orchestra.run.smithi139:> sudo egrep '\[WRN\]' /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T06:37:56.596 INFO:teuthology.orchestra.run.smithi139.stdout:2024-02-10T05:56:17.726978+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-10T06:37:56.596 INFO:tasks.cephadm:Compressing logs... 2024-02-10T06:37:56.597 DEBUG:teuthology.orchestra.run.smithi139:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T06:37:56.639 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T06:37:56.663 INFO:teuthology.orchestra.run.smithi139.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T06:37:56.664 INFO:teuthology.orchestra.run.smithi139.stderr:: No such file or directory 2024-02-10T06:37:56.665 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mon.a.log 2024-02-10T06:37:56.665 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi139.kpyuou.log 2024-02-10T06:37:56.665 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T06:37:56.665 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi139.kpyuou.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T06:37:56.667 INFO:teuthology.orchestra.run.smithi139.stderr: 76.2% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi139.kpyuou.log.gz 2024-02-10T06:37:56.667 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.3.log 2024-02-10T06:37:56.667 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mgr.y.log 2024-02-10T06:37:56.667 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.1.log: /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/tcmu-runner.log 2024-02-10T06:37:56.667 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T06:37:56.667 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T06:37:56.667 INFO:teuthology.orchestra.run.smithi139.stderr: 89.5% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/tcmu-runner.log.gz 2024-02-10T06:37:56.669 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T06:37:56.669 INFO:teuthology.orchestra.run.smithi150.stderr:: No such file or directory 2024-02-10T06:37:56.669 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.6.log 2024-02-10T06:37:56.669 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T06:37:56.669 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.4.log 2024-02-10T06:37:56.669 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.7.log 2024-02-10T06:37:56.672 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.5.log 2024-02-10T06:37:56.672 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.4.log: /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mgr.x.log 2024-02-10T06:37:56.672 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T06:37:56.673 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.log 2024-02-10T06:37:56.673 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log: 90.4% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T06:37:56.673 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T06:37:56.675 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.log: 91.0% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T06:37:56.675 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mon.b.log 2024-02-10T06:37:56.676 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-volume.log: 89.0% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T06:37:56.677 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi150.subshb.log 2024-02-10T06:37:56.677 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.0.log: 94.4% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T06:37:56.679 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.log 2024-02-10T06:37:56.680 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T06:37:56.681 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mon.b.log: 86.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T06:37:56.681 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.log: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T06:37:56.682 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi150.subshb.log: 76.3% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi150.subshb.log.gz 2024-02-10T06:37:56.683 INFO:teuthology.orchestra.run.smithi139.stderr: 94.1% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T06:37:56.683 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T06:37:56.683 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mon.c.log 2024-02-10T06:37:56.688 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-volume.log: 93.8% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T06:37:56.691 INFO:teuthology.orchestra.run.smithi150.stderr: 91.1% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T06:37:56.698 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mon.c.log: 91.1% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T06:37:56.770 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mgr.x.log.gz 2024-02-10T06:37:57.168 INFO:teuthology.orchestra.run.smithi139.stderr: 89.0% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mgr.y.log.gz 2024-02-10T06:37:57.352 INFO:teuthology.orchestra.run.smithi150.stderr: 92.3% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mon.b.log.gz 2024-02-10T06:37:57.586 INFO:teuthology.orchestra.run.smithi139.stderr: 92.4% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mon.c.log.gz 2024-02-10T06:37:58.059 INFO:teuthology.orchestra.run.smithi139.stderr: 93.2% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T06:37:58.424 INFO:teuthology.orchestra.run.smithi150.stderr: 93.0% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.6.log.gz 2024-02-10T06:37:58.576 INFO:teuthology.orchestra.run.smithi150.stderr: 93.3% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.5.log.gz 2024-02-10T06:37:58.577 INFO:teuthology.orchestra.run.smithi139.stderr: 91.1% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-mon.a.log.gz 2024-02-10T06:37:58.585 INFO:teuthology.orchestra.run.smithi139.stderr: 93.4% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T06:37:58.706 INFO:teuthology.orchestra.run.smithi150.stderr: 93.4% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.7.log.gz 2024-02-10T06:37:58.962 INFO:teuthology.orchestra.run.smithi139.stderr: 93.3% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T06:37:58.990 INFO:teuthology.orchestra.run.smithi150.stderr: 93.4% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.4.log.gz 2024-02-10T06:37:58.991 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-02-10T06:37:58.992 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m2.338s 2024-02-10T06:37:58.992 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m8.520s 2024-02-10T06:37:58.992 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m0.344s 2024-02-10T06:37:59.059 INFO:teuthology.orchestra.run.smithi139.stderr: 93.4% -- replaced with /var/log/ceph/ce44e742-c7d8-11ee-95b8-87774f69a715/ceph-osd.3.log.gz 2024-02-10T06:37:59.060 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-02-10T06:37:59.060 INFO:teuthology.orchestra.run.smithi139.stderr:real 0m2.409s 2024-02-10T06:37:59.060 INFO:teuthology.orchestra.run.smithi139.stderr:user 0m10.959s 2024-02-10T06:37:59.061 INFO:teuthology.orchestra.run.smithi139.stderr:sys 0m0.404s 2024-02-10T06:37:59.061 INFO:tasks.cephadm:Archiving logs... 2024-02-10T06:37:59.061 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103/remote/smithi139/log 2024-02-10T06:37:59.062 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T06:38:00.651 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103/remote/smithi150/log 2024-02-10T06:38:00.652 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T06:38:01.925 INFO:tasks.cephadm:Removing cluster... 2024-02-10T06:38:01.926 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 --force 2024-02-10T06:38:02.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ce44e742-c7d8-11ee-95b8-87774f69a715 --force 2024-02-10T06:38:02.513 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T06:38:02.513 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T06:38:02.528 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T06:38:02.543 INFO:tasks.cephadm:Teardown complete 2024-02-10T06:38:02.544 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T06:38:02.553 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T06:38:02.554 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:38:02.571 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:38:02.582 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:38:02.586 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:38:02.587 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-02-10T06:38:02.587 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 +623us[ +631us] +/- 81ms 2024-02-10T06:38:02.587 INFO:teuthology.orchestra.run.smithi139.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -1392us[-1384us] +/- 79ms 2024-02-10T06:38:02.587 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 +821us[ +828us] +/- 81ms 2024-02-10T06:38:02.587 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:38:02.597 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:38:02.601 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:38:02.601 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-02-10T06:38:02.601 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 101 +642us[ +666us] +/- 80ms 2024-02-10T06:38:02.601 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 -1384us[-1359us] +/- 79ms 2024-02-10T06:38:02.601 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +832us[ +857us] +/- 81ms 2024-02-10T06:38:02.601 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:38:02.602 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T06:38:02.610 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T06:38:02.611 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T06:38:02.626 DEBUG:teuthology.orchestra.run.smithi139:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T06:38:02.718 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T06:38:02.798 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T06:38:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:38:02.829 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T06:38:02.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:38:02.861 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T06:38:02.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T06:38:02.882 INFO:teuthology.task.internal:Duration was 2886.002522 seconds 2024-02-10T06:38:02.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T06:38:02.896 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T06:38:02.896 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T06:38:02.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T06:38:02.939 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T06:38:02.945 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T06:38:03.376 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T06:38:03.376 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2024-02-10T06:38:03.377 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T06:38:03.437 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-02-10T06:38:03.437 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T06:38:03.467 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T06:38:03.467 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T06:38:03.480 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T06:38:03.543 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T06:38:03.543 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T06:38:03.550 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T06:38:04.125 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T06:38:04.135 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T06:38:04.136 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T06:38:04.163 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T06:38:04.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T06:38:04.200 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:38:04.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:38:04.228 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2024-02-10T06:38:04.256 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-02-10T06:38:04.293 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:38:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:38:04.312 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:38:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:38:04.347 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T06:38:04.357 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T06:38:04.358 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103/remote/smithi139 2024-02-10T06:38:04.358 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T06:38:04.409 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103/remote/smithi150 2024-02-10T06:38:04.409 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T06:38:04.451 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T06:38:04.451 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T06:38:04.454 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T06:38:04.500 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T06:38:04.511 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T06:38:04.511 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T06:38:04.526 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T06:38:04.526 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T06:38:04.528 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T06:38:04.540 INFO:teuthology.orchestra.run.smithi139.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 06:38 /home/ubuntu/cephtest 2024-02-10T06:38:04.554 INFO:teuthology.orchestra.run.smithi150.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 06:38 /home/ubuntu/cephtest 2024-02-10T06:38:04.555 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T06:38:04.568 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T06:38:04.611 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T06:38:04.635 DEBUG:teuthology.nuke:shortname: smithi139 2024-02-10T06:38:04.635 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T06:38:04.658 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:40:12.340760', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDzupiDa8iCoDpVbGnF1CjaQKdhJ8u3hRFiv7+qev9jsLUM5KYuhRkDsNisU3v+38KyhuRC5TJmQviJeqfeSbc0='} 2024-02-10T06:38:04.661 DEBUG:teuthology.nuke:shortname: smithi150 2024-02-10T06:38:04.661 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T06:38:04.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555103', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:40:12.342075', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGPhfqC7uuXe2tG6OI42hqF3SJGQ+KtDKznYuzD3pOZLwZUoxXUsmIM4CJWmLt6gwiQgxGYdMOHJ+knZoP4uV7c='} 2024-02-10T06:38:04.707 INFO:teuthology.orchestra.console.smithi139:Power off 2024-02-10T06:38:04.708 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T06:38:04.730 DEBUG:teuthology.orchestra.console.smithi139:power off output: Chassis Power Control: Down/Off 2024-02-10T06:38:04.731 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:38:04.746 INFO:teuthology.orchestra.console.smithi150:Power off 2024-02-10T06:38:04.746 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T06:38:04.758 DEBUG:teuthology.orchestra.console.smithi139:check power output: Chassis Power is on 2024-02-10T06:38:04.770 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-02-10T06:38:04.770 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:38:04.796 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-02-10T06:38:08.760 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:38:08.797 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:38:08.871 DEBUG:teuthology.orchestra.console.smithi139:check power output: Chassis Power is on 2024-02-10T06:38:08.906 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-02-10T06:38:12.876 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:38:12.908 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:38:12.987 DEBUG:teuthology.orchestra.console.smithi139:check power output: Chassis Power is off 2024-02-10T06:38:13.024 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-02-10T06:38:13.089 INFO:teuthology.orchestra.console.smithi139:Power off completed 2024-02-10T06:38:13.125 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-02-10T06:38:13.427 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 2886.0025222301483 failure_reason: '"2024-02-10T05:56:17.726978+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' owner: scheduled_yuriw@teuthology success: false 2024-02-10T06:38:13.427 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T06:38:13.523 INFO:teuthology.run:FAIL