2024-05-11T17:55:32.326 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-11T17:55:32.326 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi --archive /home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525 --description orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} --owner scheduled_adking@teuthology -- /home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525/orig.config.yaml 2024-05-11T17:55:32.373 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-11T17:55:32.453 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525 branch: wip-adk-testing-2024-05-10-1625 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} email: adking@redhat.com first_in_suite: false job_id: '7702525' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-05-10-1625 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON log-only-match: - CEPHADM_ sha1: bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d workunit: branch: wip-adk-testing-2024-05-10-1625 sha1: bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 1272 sha1: bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d sleep_before_teardown: 0 subset: 1/20 suite: orch:cephadm suite_branch: wip-adk-testing-2024-05-10-1625 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d targets: smithi044.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEEys4usNeZqnyyizNgtpnTTuKdeUwlcF3HVdv426r+G4+HyhS7pGckEteU+20uRvl/HpEzhzNMqltPQpD+ulMo= smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBrq2hnqrZLO9TGqrGty7x++2ljkeMfclJZgbr01L8+wWM2wo4MLcdq4t+KbbZvoDRMLfs1ZdSjR+sff2ff1T8c= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-11_17:17:13 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-11T17:55:32.453 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d/qa; will attempt to use it 2024-05-11T17:55:32.454 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d/qa/tasks 2024-05-11T17:55:32.454 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-11T17:55:32.456 INFO:teuthology.task.internal:Checking packages... 2024-05-11T17:55:32.480 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d' 2024-05-11T17:55:32.481 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-11T17:55:32.481 INFO:teuthology.packaging:ref: None 2024-05-11T17:55:32.481 INFO:teuthology.packaging:tag: None 2024-05-11T17:55:32.481 INFO:teuthology.packaging:branch: wip-adk-testing-2024-05-10-1625 2024-05-11T17:55:32.481 INFO:teuthology.packaging:sha1: bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d 2024-05-11T17:55:32.481 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-05-10-1625 2024-05-11T17:55:32.700 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3686.gbfa64d6a 2024-05-11T17:55:32.701 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-11T17:55:32.711 INFO:teuthology.task.internal:no buildpackages task found 2024-05-11T17:55:32.712 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-11T17:55:32.751 INFO:teuthology.task.internal:Saving configuration 2024-05-11T17:55:32.771 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-11T17:55:32.784 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-11T17:55:32.805 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-11 17:50:38.097447', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEEys4usNeZqnyyizNgtpnTTuKdeUwlcF3HVdv426r+G4+HyhS7pGckEteU+20uRvl/HpEzhzNMqltPQpD+ulMo='} 2024-05-11T17:55:32.825 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-11 17:50:38.098640', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBrq2hnqrZLO9TGqrGty7x++2ljkeMfclJZgbr01L8+wWM2wo4MLcdq4t+KbbZvoDRMLfs1ZdSjR+sff2ff1T8c='} 2024-05-11T17:55:32.825 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-11T17:55:32.834 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-05-11T17:55:32.834 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-05-11T17:55:32.834 INFO:teuthology.run_tasks:Running task console_log... 2024-05-11T17:55:32.913 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f957225b160>, signals=[15]) 2024-05-11T17:55:32.914 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-11T17:55:32.920 INFO:teuthology.task.internal:Opening connections... 2024-05-11T17:55:32.920 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi044.front.sepia.ceph.com 2024-05-11T17:55:32.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-11T17:55:32.999 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2024-05-11T17:55:33.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-11T17:55:33.074 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-11T17:55:33.082 DEBUG:teuthology.orchestra.run.smithi044:> uname -m 2024-05-11T17:55:33.101 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2024-05-11T17:55:33.102 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream" 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="9" 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos" 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora" 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="9" 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el9" 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:LOGO="fedora-logo-icon" 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/" 2024-05-11T17:55:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-11T17:55:33.161 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-11T17:55:33.161 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-11T17:55:33.161 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2024-05-11T17:55:33.217 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2024-05-11T17:55:33.234 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2024-05-11T17:55:33.234 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2024-05-11T17:55:33.290 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream" 2024-05-11T17:55:33.290 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9" 2024-05-11T17:55:33.290 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos" 2024-05-11T17:55:33.290 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora" 2024-05-11T17:55:33.290 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9" 2024-05-11T17:55:33.290 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9" 2024-05-11T17:55:33.290 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-11T17:55:33.290 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2024-05-11T17:55:33.290 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon" 2024-05-11T17:55:33.291 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-11T17:55:33.291 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/" 2024-05-11T17:55:33.291 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-11T17:55:33.291 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-11T17:55:33.291 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-11T17:55:33.291 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2024-05-11T17:55:33.307 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-11T17:55:33.318 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-11T17:55:33.388 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-11T17:55:33.388 DEBUG:teuthology.orchestra.run.smithi044:> test '!' -e /home/ubuntu/cephtest 2024-05-11T17:55:33.390 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2024-05-11T17:55:33.406 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-11T17:55:33.416 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-11T17:55:33.416 DEBUG:teuthology.orchestra.run.smithi044:> test -z $(ls -A /var/lib/ceph) 2024-05-11T17:55:33.448 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2024-05-11T17:55:33.476 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-11T17:55:33.539 INFO:teuthology.run_tasks:Running task kernel... 2024-05-11T17:55:33.560 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-11T17:55:33.560 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-11T17:55:33.560 DEBUG:teuthology.orchestra.run.smithi044:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-11T17:55:33.561 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-11T17:55:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-11T17:55:33.577 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-05-11T17:55:33.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-11T17:55:33.578 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-05-11T17:55:33.634 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-437.el9.x86_64 2024-05-11T17:55:33.634 INFO:teuthology.task.kernel:Running kernel on smithi092: 5.14.0-437.el9.x86_64 2024-05-11T17:55:33.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum install -y kernel 2024-05-11T17:55:33.636 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-437.el9.x86_64 2024-05-11T17:55:33.636 INFO:teuthology.task.kernel:Running kernel on smithi044: 5.14.0-437.el9.x86_64 2024-05-11T17:55:33.636 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum install -y kernel 2024-05-11T17:55:35.803 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.1 MB 00:00 2024-05-11T17:55:36.106 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.1 MB 00:00 2024-05-11T17:55:37.956 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-05-11T17:55:39.029 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 19 MB 00:02 2024-05-11T17:55:41.898 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.2 MB 00:00 2024-05-11T17:55:43.230 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-05-11T17:55:43.626 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-05-11T17:55:44.739 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 16 kB 00:00 2024-05-11T17:55:47.309 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 21 MB 00:02 2024-05-11T17:55:47.477 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 5.7 MB/s | 21 MB 00:03 2024-05-11T17:55:52.666 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-11T17:55:52.707 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-05-11T17:55:54.714 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-11T17:55:54.714 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-11T17:55:54.763 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-11T17:55:54.763 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-11T17:55:54.814 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-05-11T17:55:54.825 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-05-11T17:55:54.825 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2024-05-11T17:55:54.825 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-05-11T17:55:54.825 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-05-11T17:55:54.826 INFO:teuthology.orchestra.run.smithi092.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-11T17:55:54.826 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-05-11T17:55:54.826 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-11T17:55:54.826 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-11T17:55:54.826 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-11T17:55:54.826 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-05-11T17:55:54.826 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-05-11T17:55:54.826 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-05-11T17:55:54.826 INFO:teuthology.orchestra.run.smithi092.stdout:Install 4 Packages 2024-05-11T17:55:54.826 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-05-11T17:55:54.827 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 78 M 2024-05-11T17:55:54.827 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 123 M 2024-05-11T17:55:54.827 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-05-11T17:55:54.854 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-05-11T17:55:54.867 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-05-11T17:55:54.867 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-05-11T17:55:54.867 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-05-11T17:55:54.867 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-05-11T17:55:54.868 INFO:teuthology.orchestra.run.smithi044.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-11T17:55:54.868 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-05-11T17:55:54.868 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-11T17:55:54.868 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-11T17:55:54.868 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-11T17:55:54.868 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-11T17:55:54.868 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-05-11T17:55:54.868 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-05-11T17:55:54.868 INFO:teuthology.orchestra.run.smithi044.stdout:Install 4 Packages 2024-05-11T17:55:54.868 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-11T17:55:54.869 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 78 M 2024-05-11T17:55:54.869 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 123 M 2024-05-11T17:55:54.869 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-05-11T17:55:55.192 INFO:teuthology.orchestra.run.smithi044.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 1.8 MB/s | 484 kB 00:00 2024-05-11T17:55:55.434 INFO:teuthology.orchestra.run.smithi092.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 1.4 MB/s | 484 kB 00:00 2024-05-11T17:55:57.934 INFO:teuthology.orchestra.run.smithi092.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 5.4 MB/s | 15 MB 00:02 2024-05-11T17:55:58.792 INFO:teuthology.orchestra.run.smithi044.stdout:(2/4): kernel-modules-core-5.14.0-444.el9.x86_6 7.8 MB/s | 28 MB 00:03 2024-05-11T17:55:59.651 INFO:teuthology.orchestra.run.smithi044.stdout:(3/4): kernel-core-5.14.0-444.el9.x86_64.rpm 3.3 MB/s | 15 MB 00:04 2024-05-11T17:56:00.217 INFO:teuthology.orchestra.run.smithi092.stdout:(3/4): kernel-modules-core-5.14.0-444.el9.x86_6 5.8 MB/s | 28 MB 00:04 2024-05-11T17:56:00.317 INFO:teuthology.orchestra.run.smithi044.stdout:(4/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 6.3 MB/s | 34 MB 00:05 2024-05-11T17:56:00.318 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-05-11T17:56:00.318 INFO:teuthology.orchestra.run.smithi044.stdout:Total 14 MB/s | 78 MB 00:05 2024-05-11T17:56:00.900 INFO:teuthology.orchestra.run.smithi092.stdout:(4/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 5.8 MB/s | 34 MB 00:05 2024-05-11T17:56:00.900 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-05-11T17:56:00.901 INFO:teuthology.orchestra.run.smithi092.stdout:Total 13 MB/s | 78 MB 00:06 2024-05-11T17:56:01.024 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-05-11T17:56:01.410 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-05-11T17:56:01.411 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-05-11T17:56:01.581 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-05-11T17:56:01.934 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-05-11T17:56:01.935 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-05-11T17:56:02.361 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-05-11T17:56:02.361 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-05-11T17:56:02.837 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-05-11T17:56:02.838 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-05-11T17:56:04.895 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-05-11T17:56:05.252 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-05-11T17:56:05.550 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-11T17:56:05.642 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-11T17:56:05.885 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-11T17:56:05.991 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-11T17:56:08.438 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-11T17:56:08.563 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-11T17:56:08.653 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-11T17:56:08.785 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-11T17:56:13.500 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-11T17:56:13.620 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:13.638 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-11T17:56:13.683 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:18.332 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:18.772 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:48.127 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:48.127 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-11T17:56:48.127 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-11T17:56:48.127 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-11T17:56:48.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:48.896 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-05-11T17:56:48.896 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-05-11T17:56:48.896 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-11T17:56:48.896 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-11T17:56:48.896 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-11T17:56:48.896 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-11T17:56:48.896 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-05-11T17:56:48.896 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-05-11T17:56:49.243 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-05-11T17:56:49.334 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-11T17:56:49.335 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-11T17:56:49.335 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-11T17:56:49.335 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-11T17:56:49.335 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-11T17:56:49.335 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-11T17:56:49.335 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2024-05-11T17:56:49.335 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-05-11T17:56:49.385 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-11T17:56:49.386 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-11T17:56:49.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -qi grub2-tools 2024-05-11T17:56:49.426 INFO:teuthology.orchestra.run.smithi092.stdout:Name : grub2-tools 2024-05-11T17:56:49.426 INFO:teuthology.orchestra.run.smithi092.stdout:Epoch : 1 2024-05-11T17:56:49.426 INFO:teuthology.orchestra.run.smithi092.stdout:Version : 2.06 2024-05-11T17:56:49.426 INFO:teuthology.orchestra.run.smithi092.stdout:Release : 68.el9 2024-05-11T17:56:49.426 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: x86_64 2024-05-11T17:56:49.426 INFO:teuthology.orchestra.run.smithi092.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-11T17:56:49.426 INFO:teuthology.orchestra.run.smithi092.stdout:Group : Unspecified 2024-05-11T17:56:49.426 INFO:teuthology.orchestra.run.smithi092.stdout:Size : 8264254 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout:License : GPLv3+ 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout:Packager : builder@centos.org 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout:Vendor : CentOS 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout:Summary : Support tools for GRUB. 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout:Description : 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-05-11T17:56:49.427 INFO:teuthology.orchestra.run.smithi092.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-11T17:56:49.428 INFO:teuthology.orchestra.run.smithi092.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-11T17:56:49.428 INFO:teuthology.orchestra.run.smithi092.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-11T17:56:49.428 INFO:teuthology.orchestra.run.smithi092.stdout:hardware devices. 2024-05-11T17:56:49.428 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-05-11T17:56:49.428 INFO:teuthology.orchestra.run.smithi092.stdout:This subpackage provides tools for support of all platforms. 2024-05-11T17:56:49.429 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-11T17:56:49.429 INFO:teuthology.task.kernel:Updating grub on smithi092 to boot 5.14.0-444.el9.x86_64 2024-05-11T17:56:49.429 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-11T17:56:49.848 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:49.887 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:50.053 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2024-05-11T17:56:50.265 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:50.265 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-11T17:56:50.265 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-11T17:56:50.265 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-11T17:56:51.059 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-11T17:56:51.060 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-11T17:56:51.060 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-05-11T17:56:51.060 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-11T17:56:51.060 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-11T17:56:51.060 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-11T17:56:51.060 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-11T17:56:51.060 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-11T17:56:51.060 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-05-11T17:56:51.350 INFO:teuthology.orchestra.run.smithi092.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-11T17:56:51.380 INFO:teuthology.orchestra.run.smithi092.stderr:done 2024-05-11T17:56:51.382 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-05-11T17:56:51.395 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2024-05-11T17:56:51.398 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.sfJyKaeKbV 2024-05-11T17:56:51.399 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sfJyKaeKbV 2024-05-11T17:56:51.464 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-11T17:56:51.465 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-11T17:56:51.465 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-11T17:56:51.465 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-11T17:56:51.465 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-11T17:56:51.465 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-11T17:56:51.465 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi044.front.sepia.ceph.com, path=None, version=distro) 2024-05-11T17:56:51.465 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2024-05-11T17:56:51.467 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.sfJyKaeKbV 2024-05-11T17:56:51.515 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-11T17:56:51.516 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-11T17:56:51.516 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -qi grub2-tools 2024-05-11T17:56:51.553 INFO:teuthology.orchestra.run.smithi044.stdout:Name : grub2-tools 2024-05-11T17:56:51.553 INFO:teuthology.orchestra.run.smithi044.stdout:Epoch : 1 2024-05-11T17:56:51.553 INFO:teuthology.orchestra.run.smithi044.stdout:Version : 2.06 2024-05-11T17:56:51.553 INFO:teuthology.orchestra.run.smithi044.stdout:Release : 68.el9 2024-05-11T17:56:51.553 INFO:teuthology.orchestra.run.smithi044.stdout:Architecture: x86_64 2024-05-11T17:56:51.553 INFO:teuthology.orchestra.run.smithi044.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-11T17:56:51.553 INFO:teuthology.orchestra.run.smithi044.stdout:Group : Unspecified 2024-05-11T17:56:51.553 INFO:teuthology.orchestra.run.smithi044.stdout:Size : 8264254 2024-05-11T17:56:51.553 INFO:teuthology.orchestra.run.smithi044.stdout:License : GPLv3+ 2024-05-11T17:56:51.553 INFO:teuthology.orchestra.run.smithi044.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-11T17:56:51.553 INFO:teuthology.orchestra.run.smithi044.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout:Packager : builder@centos.org 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout:Vendor : CentOS 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout:Summary : Support tools for GRUB. 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout:Description : 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout:hardware devices. 2024-05-11T17:56:51.554 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-11T17:56:51.555 INFO:teuthology.orchestra.run.smithi044.stdout:This subpackage provides tools for support of all platforms. 2024-05-11T17:56:51.555 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-11T17:56:51.555 INFO:teuthology.task.kernel:Updating grub on smithi044 to boot 5.14.0-444.el9.x86_64 2024-05-11T17:56:51.556 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-11T17:56:51.608 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.sfJyKaeKbV is 6KB 2024-05-11T17:56:51.657 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.sfJyKaeKbV 2024-05-11T17:56:51.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo /bin/ls /boot/loader/entries || true 2024-05-11T17:56:51.742 INFO:teuthology.orchestra.run.smithi092.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-11T17:56:51.743 INFO:teuthology.orchestra.run.smithi092.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-11T17:56:51.743 INFO:teuthology.orchestra.run.smithi092.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-11T17:56:51.743 INFO:teuthology.orchestra.run.smithi092.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-11T17:56:51.744 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-11T17:56:51.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2024-05-11T17:56:52.190 INFO:teuthology.orchestra.run.smithi044.stderr:Generating grub configuration file ... 2024-05-11T17:56:53.518 INFO:teuthology.orchestra.run.smithi044.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-11T17:56:53.553 INFO:teuthology.orchestra.run.smithi044.stderr:done 2024-05-11T17:56:53.555 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-05-11T17:56:53.570 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.lEfZI4bUvg 2024-05-11T17:56:53.570 DEBUG:teuthology.orchestra.run.smithi044:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lEfZI4bUvg 2024-05-11T17:56:53.636 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.lEfZI4bUvg 2024-05-11T17:56:53.780 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.lEfZI4bUvg is 6KB 2024-05-11T17:56:53.830 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.lEfZI4bUvg 2024-05-11T17:56:53.844 DEBUG:teuthology.orchestra.run.smithi044:> sudo /bin/ls /boot/loader/entries || true 2024-05-11T17:56:53.906 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-11T17:56:53.906 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-11T17:56:53.906 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-11T17:56:53.906 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-11T17:56:53.908 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-11T17:56:54.062 DEBUG:teuthology.orchestra.run.smithi044:> sudo shutdown -r now 2024-05-11T17:57:21.910 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-11T17:57:21.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-05-11T17:57:21.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-11T17:57:24.067 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-11T17:57:24.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-11T17:57:24.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-11T17:57:40.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-05-11T17:57:42.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.44 2024-05-11T17:57:49.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-05-11T17:57:49.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-11T17:57:51.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-11T17:57:51.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-11T17:58:49.317 DEBUG:teuthology.orchestra.remote:timed out 2024-05-11T17:58:51.566 DEBUG:teuthology.orchestra.remote:timed out 2024-05-11T17:59:01.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-05-11T17:59:01.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-11T17:59:01.762 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-05-11T17:59:02.130 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-05-11T17:59:02.130 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-444.el9.x86_64"... 2024-05-11T17:59:02.130 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-05-11T17:59:02.149 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-444.el9.x86_64 2024-05-11T17:59:02.149 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-11T17:59:02.149 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-11T17:59:02.150 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-11T17:59:03.151 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-11T17:59:03.151 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-11T17:59:03.216 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2024-05-11T17:59:03.247 DEBUG:teuthology.parallel:result is None 2024-05-11T17:59:03.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-11T17:59:03.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-11T17:59:03.836 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-05-11T17:59:04.230 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-11T17:59:04.230 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-444.el9.x86_64"... 2024-05-11T17:59:04.230 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-05-11T17:59:04.248 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-444.el9.x86_64 2024-05-11T17:59:04.248 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-11T17:59:04.248 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-11T17:59:04.248 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-11T17:59:05.249 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-11T17:59:05.250 DEBUG:teuthology.orchestra.run.smithi044:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-11T17:59:05.318 INFO:teuthology.orchestra.run.smithi044.stdout:ttyS1 2024-05-11T17:59:05.347 DEBUG:teuthology.parallel:result is None 2024-05-11T17:59:05.347 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-11T17:59:05.355 INFO:teuthology.task.internal:Creating test directory... 2024-05-11T17:59:05.356 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-11T17:59:05.358 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-11T17:59:05.376 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-11T17:59:05.383 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-11T17:59:05.394 INFO:teuthology.task.internal:Creating archive directory... 2024-05-11T17:59:05.394 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-11T17:59:05.417 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-11T17:59:05.456 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-11T17:59:05.462 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-11T17:59:05.463 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-11T17:59:05.486 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-11T17:59:05.515 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-11T17:59:05.520 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-11T17:59:05.526 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-11T17:59:05.531 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-11T17:59:05.533 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-11T17:59:05.542 INFO:teuthology.task.internal:Configuring sudo... 2024-05-11T17:59:05.542 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-11T17:59:05.571 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-11T17:59:05.600 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-11T17:59:05.610 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-11T17:59:05.610 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-11T17:59:05.640 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-11T17:59:05.657 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-11T17:59:05.741 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-11T17:59:05.783 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-11T17:59:05.867 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-11T17:59:05.909 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-11T17:59:05.909 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-11T17:59:05.976 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-11T17:59:06.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-11T17:59:06.074 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-11T17:59:06.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-11T17:59:06.201 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-05-11T17:59:06.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-11T17:59:06.266 DEBUG:teuthology.orchestra.run.smithi044:> sudo service rsyslog restart 2024-05-11T17:59:06.269 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2024-05-11T17:59:06.311 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-11T17:59:06.358 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-11T17:59:06.705 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-11T17:59:06.713 INFO:teuthology.task.internal:Starting timer... 2024-05-11T17:59:06.714 INFO:teuthology.run_tasks:Running task pcp... 2024-05-11T17:59:06.724 INFO:teuthology.run_tasks:Running task selinux... 2024-05-11T17:59:06.733 DEBUG:teuthology.orchestra.run.smithi044:> sudo service auditd rotate 2024-05-11T17:59:06.786 INFO:teuthology.orchestra.run.smithi044.stdout:Rotating logs: 2024-05-11T17:59:06.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2024-05-11T17:59:06.875 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2024-05-11T17:59:06.877 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-11T17:59:06.877 DEBUG:teuthology.orchestra.run.smithi044:> /usr/sbin/getenforce 2024-05-11T17:59:06.908 INFO:teuthology.orchestra.run.smithi044.stdout:Permissive 2024-05-11T17:59:06.908 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2024-05-11T17:59:06.976 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2024-05-11T17:59:06.977 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi044.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi092.front.sepia.ceph.com': 'permissive'} 2024-05-11T17:59:06.977 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-11T17:59:07.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-11T17:59:07.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-11T17:59:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-11T17:59:07.032 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-11T17:59:07.032 DEBUG:teuthology.orchestra.run.smithi044:> sudo /usr/sbin/setenforce permissive 2024-05-11T17:59:07.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2024-05-11T17:59:07.111 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-11T17:59:07.121 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-11T17:59:07.140 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-11T17:59:07.141 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi044.front.sepia.ceph.com,smithi092.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-11T18:03:10.164 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi044.front.sepia.ceph.com'), Remote(name='ubuntu@smithi092.front.sepia.ceph.com')] 2024-05-11T18:03:10.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-11T18:03:10.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-11T18:03:10.297 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-05-11T18:03:10.387 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-11T18:03:10.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-05-11T18:03:10.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-11T18:03:10.464 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-05-11T18:03:10.537 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-05-11T18:03:10.538 INFO:teuthology.run_tasks:Running task clock... 2024-05-11T18:03:10.550 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-11T18:03:10.550 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-11T18:03:10.550 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-11T18:03:10.553 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-11T18:03:10.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-11T18:03:10.584 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-11T18:03:10.599 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-11T18:03:10.605 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-11T18:03:10.619 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-11T18:03:10.626 INFO:teuthology.orchestra.run.smithi044.stderr:sudo: ntpd: command not found 2024-05-11T18:03:10.637 INFO:teuthology.orchestra.run.smithi044.stdout:506 Cannot talk to daemon 2024-05-11T18:03:10.643 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2024-05-11T18:03:10.653 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-11T18:03:10.654 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2024-05-11T18:03:10.667 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-11T18:03:10.668 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-11T18:03:10.682 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-11T18:03:10.724 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-05-11T18:03:10.729 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-11T18:03:10.729 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-05-11T18:03:10.729 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-11T18:03:10.729 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-11T18:03:10.729 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-11T18:03:10.729 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-11T18:03:10.736 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-05-11T18:03:10.740 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-11T18:03:10.740 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-05-11T18:03:10.740 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-11T18:03:10.740 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-11T18:03:10.740 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-11T18:03:10.741 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-11T18:03:10.741 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-11T18:03:10.854 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON'], 'log-only-match': ['CEPHADM_'], 'sha1': 'bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d'} 2024-05-11T18:03:10.854 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-05-11T18:03:10.855 INFO:tasks.cephadm:Cluster fsid is ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:03:10.855 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-11T18:03:10.855 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.44', 'mon.c': '[v2:172.21.15.44:3301,v1:172.21.15.44:6790]', 'mon.b': '172.21.15.92'} 2024-05-11T18:03:10.855 INFO:tasks.cephadm:First mon is mon.a on smithi044 2024-05-11T18:03:10.855 INFO:tasks.cephadm:First mgr is y 2024-05-11T18:03:10.855 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-11T18:03:10.855 DEBUG:teuthology.orchestra.run.smithi044:> sudo hostname $(hostname -s) 2024-05-11T18:03:10.883 DEBUG:teuthology.orchestra.run.smithi092:> sudo hostname $(hostname -s) 2024-05-11T18:03:10.909 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-05-11T18:03:10.909 DEBUG:teuthology.orchestra.run.smithi044:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-11T18:03:11.155 INFO:teuthology.orchestra.run.smithi044.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 May 11 18:03 /home/ubuntu/cephtest/cephadm 2024-05-11T18:03:11.155 DEBUG:teuthology.orchestra.run.smithi092:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-11T18:03:11.318 INFO:teuthology.orchestra.run.smithi092.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 May 11 18:03 /home/ubuntu/cephtest/cephadm 2024-05-11T18:03:11.319 DEBUG:teuthology.orchestra.run.smithi044:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-11T18:03:11.336 DEBUG:teuthology.orchestra.run.smithi092:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-11T18:03:11.362 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-05-11T18:03:11.362 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-05-11T18:03:11.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-05-11T18:03:12.981 INFO:teuthology.orchestra.run.smithi092.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-11T18:03:13.065 INFO:teuthology.orchestra.run.smithi044.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-11T18:03:30.346 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-05-11T18:03:30.347 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-05-11T18:03:30.347 INFO:teuthology.orchestra.run.smithi092.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-05-11T18:03:30.347 INFO:teuthology.orchestra.run.smithi092.stdout: "repo_digests": [ 2024-05-11T18:03:30.347 INFO:teuthology.orchestra.run.smithi092.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-05-11T18:03:30.347 INFO:teuthology.orchestra.run.smithi092.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-05-11T18:03:30.347 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-05-11T18:03:30.347 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-05-11T18:03:31.269 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-11T18:03:31.269 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-05-11T18:03:31.269 INFO:teuthology.orchestra.run.smithi044.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-05-11T18:03:31.269 INFO:teuthology.orchestra.run.smithi044.stdout: "repo_digests": [ 2024-05-11T18:03:31.269 INFO:teuthology.orchestra.run.smithi044.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-05-11T18:03:31.269 INFO:teuthology.orchestra.run.smithi044.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-05-11T18:03:31.270 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-11T18:03:31.270 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-11T18:03:31.289 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph 2024-05-11T18:03:31.349 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph 2024-05-11T18:03:31.382 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 777 /etc/ceph 2024-05-11T18:03:31.409 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /etc/ceph 2024-05-11T18:03:31.451 INFO:tasks.cephadm:Writing seed config... 2024-05-11T18:03:31.452 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-05-11T18:03:31.452 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-11T18:03:31.452 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-11T18:03:31.452 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-05-11T18:03:31.452 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-11T18:03:31.452 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-11T18:03:31.453 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-11T18:03:31.453 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-11T18:03:31.453 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-11T18:03:31.453 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-11T18:03:31.453 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-11T18:03:31.470 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ba7b3ba0-0fc0-11ef-bc98-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-05-11T18:03:31.470 DEBUG:teuthology.orchestra.run.smithi044:mon.a> sudo journalctl -f -n 0 -u ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mon.a.service 2024-05-11T18:03:31.513 DEBUG:teuthology.orchestra.run.smithi044:mgr.y> sudo journalctl -f -n 0 -u ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mgr.y.service 2024-05-11T18:03:31.555 INFO:tasks.cephadm:Bootstrapping... 2024-05-11T18:03:31.555 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid ba7b3ba0-0fc0-11ef-bc98-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.44 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-11T18:03:31.757 INFO:teuthology.orchestra.run.smithi044.stderr:-------------------------------------------------------------------------------- 2024-05-11T18:03:31.757 INFO:teuthology.orchestra.run.smithi044.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'ba7b3ba0-0fc0-11ef-bc98-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.44', '--skip-admin-label'] 2024-05-11T18:03:31.789 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: 4.9.4-dev 2024-05-11T18:03:31.791 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying podman|docker is present... 2024-05-11T18:03:31.813 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: 4.9.4-dev 2024-05-11T18:03:31.815 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying lvm2 is present... 2024-05-11T18:03:31.815 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying time synchronization is in place... 2024-05-11T18:03:31.822 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-05-11T18:03:31.829 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-05-11T18:03:31.836 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: enabled 2024-05-11T18:03:31.842 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: active 2024-05-11T18:03:31.843 INFO:teuthology.orchestra.run.smithi044.stderr:Unit chronyd.service is enabled and running 2024-05-11T18:03:31.843 INFO:teuthology.orchestra.run.smithi044.stderr:Repeating the final host check... 2024-05-11T18:03:31.865 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: 4.9.4-dev 2024-05-11T18:03:31.867 INFO:teuthology.orchestra.run.smithi044.stderr:podman (/bin/podman) version 4.9.4 is present 2024-05-11T18:03:31.867 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl is present 2024-05-11T18:03:31.867 INFO:teuthology.orchestra.run.smithi044.stderr:lvcreate is present 2024-05-11T18:03:31.874 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-05-11T18:03:31.880 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-05-11T18:03:31.886 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: enabled 2024-05-11T18:03:31.893 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: active 2024-05-11T18:03:31.894 INFO:teuthology.orchestra.run.smithi044.stderr:Unit chronyd.service is enabled and running 2024-05-11T18:03:31.894 INFO:teuthology.orchestra.run.smithi044.stderr:Host looks OK 2024-05-11T18:03:31.894 INFO:teuthology.orchestra.run.smithi044.stderr:Cluster fsid: ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:03:31.894 INFO:teuthology.orchestra.run.smithi044.stderr:Acquiring lock 140067795898224 on /run/cephadm/ba7b3ba0-0fc0-11ef-bc98-c7b262605968.lock 2024-05-11T18:03:31.894 INFO:teuthology.orchestra.run.smithi044.stderr:Lock 140067795898224 acquired on /run/cephadm/ba7b3ba0-0fc0-11ef-bc98-c7b262605968.lock 2024-05-11T18:03:31.895 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying IP 172.21.15.44 port 3300 ... 2024-05-11T18:03:31.895 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying IP 172.21.15.44 port 6789 ... 2024-05-11T18:03:31.895 INFO:teuthology.orchestra.run.smithi044.stderr:Base mon IP is 172.21.15.44, final addrv is [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-05-11T18:03:31.897 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.44 metric 100 2024-05-11T18:03:31.897 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.44 metric 100 2024-05-11T18:03:31.900 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-05-11T18:03:31.900 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-05-11T18:03:31.900 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-05-11T18:03:31.903 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-11T18:03:31.903 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-05-11T18:03:31.903 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-05-11T18:03:31.903 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-05-11T18:03:31.903 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-05-11T18:03:31.903 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-05-11T18:03:31.904 INFO:teuthology.orchestra.run.smithi044.stderr:Mon IP `172.21.15.44` is in CIDR network `172.21.0.0/20` 2024-05-11T18:03:31.904 INFO:teuthology.orchestra.run.smithi044.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-11T18:03:31.905 INFO:teuthology.orchestra.run.smithi044.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-11T18:03:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-05-11T18:03:32.413 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Getting image source signatures 2024-05-11T18:03:32.413 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-05-11T18:03:32.414 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-05-11T18:03:32.414 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-05-11T18:03:32.414 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-05-11T18:03:32.414 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-05-11T18:03:32.414 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-05-11T18:03:32.605 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Writing manifest to image destination 2024-05-11T18:03:32.717 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-05-11T18:03:33.172 INFO:teuthology.orchestra.run.smithi044.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-05-11T18:03:36.301 INFO:teuthology.orchestra.run.smithi044.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-05-11T18:03:36.301 INFO:teuthology.orchestra.run.smithi044.stderr:Extracting ceph user uid/gid from container image... 2024-05-11T18:03:40.079 INFO:teuthology.orchestra.run.smithi044.stderr:stat: 167 167 2024-05-11T18:03:43.431 INFO:teuthology.orchestra.run.smithi044.stderr:Creating initial keys... 2024-05-11T18:03:44.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-authtool: AQAAsz9m3uHACBAA0Em4TynENF0yL0Xa+XPysQ== 2024-05-11T18:03:45.378 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-authtool: AQABsz9mo0+mERAAMAAIkwu3OvGBvbGO8Fm7fA== 2024-05-11T18:03:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-authtool: AQACsz9mUJQUBRAAXuBJ+HoD+z6Jj/AgPpo4BA== 2024-05-11T18:03:46.607 INFO:teuthology.orchestra.run.smithi044.stderr:Creating initial monmap... 2024-05-11T18:03:46.990 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-11T18:03:46.990 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-05-11T18:03:46.990 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:03:46.990 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-11T18:03:47.530 INFO:teuthology.orchestra.run.smithi044.stderr:monmaptool for a [v2:172.21.15.44:3300,v1:172.21.15.44:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-11T18:03:47.530 INFO:teuthology.orchestra.run.smithi044.stderr:setting min_mon_release = octopus 2024-05-11T18:03:47.530 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/monmaptool: set fsid to ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:03:47.530 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-11T18:03:47.530 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-05-11T18:03:47.530 INFO:teuthology.orchestra.run.smithi044.stderr:Creating mon... 2024-05-11T18:03:50.452 INFO:teuthology.orchestra.run.smithi044.stderr:create mon.a on 2024-05-11T18:03:50.670 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-11T18:03:50.825 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968.target. 2024-05-11T18:03:50.825 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968.target. 2024-05-11T18:03:51.138 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Failed to reset failed state of unit ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mon.a.service: Unit ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mon.a.service not loaded. 2024-05-11T18:03:51.146 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Created symlink /etc/systemd/system/ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968.target.wants/ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mon.a.service → /etc/systemd/system/ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@.service. 2024-05-11T18:03:51.850 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: disabled 2024-05-11T18:03:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-05-11T18:03:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:firewalld.service is not enabled 2024-05-11T18:03:51.858 INFO:teuthology.orchestra.run.smithi044.stderr:Not possible to enable service . firewalld.service is not available 2024-05-11T18:03:51.858 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mon to start... 2024-05-11T18:03:51.858 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mon... 2024-05-11T18:03:52.353 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: cluster: 2024-05-11T18:03:52.353 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: id: ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:03:52.353 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-05-11T18:03:52.353 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:03:52.353 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: services: 2024-05-11T18:03:52.353 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.573002s) 2024-05-11T18:03:52.354 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mgr: no daemons active 2024-05-11T18:03:52.354 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-05-11T18:03:52.354 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:03:52.354 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: data: 2024-05-11T18:03:52.354 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-05-11T18:03:52.354 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-05-11T18:03:52.354 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-05-11T18:03:52.354 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: pgs: 2024-05-11T18:03:52.355 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:03:52.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:52 smithi044 ceph-mon[22628]: from='client.? 172.21.15.44:0/3822567466' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-11T18:03:52.997 INFO:teuthology.orchestra.run.smithi044.stderr:mon is available 2024-05-11T18:03:52.997 INFO:teuthology.orchestra.run.smithi044.stderr:Assimilating anything we can from ceph.conf... 2024-05-11T18:03:53.491 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:03:53.491 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: [global] 2024-05-11T18:03:53.491 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: fsid = ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:03:53.491 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: [mgr] 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: [osd] 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-05-11T18:03:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-05-11T18:03:54.067 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:53 smithi044 ceph-mon[22628]: from='client.? 172.21.15.44:0/676210906' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-11T18:03:54.068 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:53 smithi044 ceph-mon[22628]: from='client.? 172.21.15.44:0/676210906' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-11T18:03:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:Generating new minimal ceph.conf... 2024-05-11T18:03:54.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:54 smithi044 ceph-mon[22628]: from='client.? 172.21.15.44:0/4100942998' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-11T18:03:55.303 INFO:teuthology.orchestra.run.smithi044.stderr:Restarting the monitor... 2024-05-11T18:03:55.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:55 smithi044 systemd[1]: Stopping Ceph mon.a for ba7b3ba0-0fc0-11ef-bc98-c7b262605968... 2024-05-11T18:03:55.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:55 smithi044 bash[22846]: Error: no container with name or ID "ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon.a" found: no such container 2024-05-11T18:03:55.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:55 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a[22624]: 2024-05-11T18:03:55.469+0000 7f217a9a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-11T18:03:55.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:55 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a[22624]: 2024-05-11T18:03:55.469+0000 7f217a9a3700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-11T18:03:55.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:55 smithi044 podman[22859]: 2024-05-11 18:03:55.665135209 +0000 UTC m=+0.296370322 container died 0ef06096e79b382142316b133dc896677b6fbf6ff93776370b5e9b50ec70b685 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, distribution-scope=public) 2024-05-11T18:03:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 podman[22859]: 2024-05-11 18:03:56.152979606 +0000 UTC m=+0.784214727 container cleanup 0ef06096e79b382142316b133dc896677b6fbf6ff93776370b5e9b50ec70b685 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.openshift.expose-services=, distribution-scope=public, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, architecture=x86_64, version=8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git) 2024-05-11T18:03:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 bash[22859]: ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a 2024-05-11T18:03:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 bash[22886]: Error: no container with name or ID "ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon.a" found: no such container 2024-05-11T18:03:56.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 podman[22873]: 2024-05-11 18:03:56.370780308 +0000 UTC m=+0.699779964 container remove 0ef06096e79b382142316b133dc896677b6fbf6ff93776370b5e9b50ec70b685 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., vcs-type=git, architecture=x86_64, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, release=754, GIT_CLEAN=True, name=centos-stream, ceph=True) 2024-05-11T18:03:56.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 systemd[1]: ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-11T18:03:56.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 systemd[1]: Stopped Ceph mon.a for ba7b3ba0-0fc0-11ef-bc98-c7b262605968. 2024-05-11T18:03:56.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 systemd[1]: Starting Ceph mon.a for ba7b3ba0-0fc0-11ef-bc98-c7b262605968... 2024-05-11T18:03:56.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 podman[22953]: 2024-05-11 18:03:56.60496125 +0000 UTC m=+0.102119973 container create 0c428b682c7b3044ba6f74a77637ef3aa54bb28503cd598c456a3a31e9838302 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.openshift.tags=base centos centos-stream, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, release=754, RELEASE=HEAD, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8) 2024-05-11T18:03:56.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 podman[22953]: 2024-05-11 18:03:56.522388953 +0000 UTC m=+0.019547676 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-05-11T18:03:56.859 INFO:teuthology.orchestra.run.smithi044.stderr:Setting mon public_network to 172.21.0.0/20 2024-05-11T18:03:56.984 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 podman[22953]: 2024-05-11 18:03:56.799058211 +0000 UTC m=+0.296216934 container init 0c428b682c7b3044ba6f74a77637ef3aa54bb28503cd598c456a3a31e9838302 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.component=centos-stream-container, ceph=True, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, architecture=x86_64, vcs-type=git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, version=8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-11T18:03:56.984 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 podman[22953]: 2024-05-11 18:03:56.802269537 +0000 UTC m=+0.299428261 container start 0c428b682c7b3044ba6f74a77637ef3aa54bb28503cd598c456a3a31e9838302 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a, io.openshift.tags=base centos centos-stream, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, distribution-scope=public, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, ceph=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8) 2024-05-11T18:03:56.984 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: set uid:gid to 167:167 (ceph:ceph) 2024-05-11T18:03:56.984 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-05-11T18:03:56.984 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: pidfile_write: ignore empty --pid-file 2024-05-11T18:03:56.984 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: load: jerasure load: lrc 2024-05-11T18:03:56.985 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: RocksDB version: 6.15.5 2024-05-11T18:03:56.985 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-05-11T18:03:56.985 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Compile date Apr 18 2022 2024-05-11T18:03:56.985 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: DB SUMMARY 2024-05-11T18:03:56.985 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: DB Session ID: 8IJUK0FFAFWN2F97P7GD 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: CURRENT file: CURRENT 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: IDENTITY file: IDENTITY 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82629 ; 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.error_if_exists: 0 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.create_if_missing: 0 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.paranoid_checks: 1 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.env: 0x557f35cc6860 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.fs: Posix File System 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.info_log: 0x557f384b9dc0 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_file_opening_threads: 16 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.statistics: (nil) 2024-05-11T18:03:56.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.use_fsync: 0 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_log_file_size: 0 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.keep_log_file_num: 1000 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.recycle_log_file_num: 0 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.allow_fallocate: 1 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.allow_mmap_reads: 0 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.allow_mmap_writes: 0 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.use_direct_reads: 0 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.create_missing_column_families: 0 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.db_log_dir: 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-11T18:03:56.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.advise_random_on_open: 1 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.db_write_buffer_size: 0 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.write_buffer_manager: 0x557f385aa2a0 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.rate_limiter: (nil) 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.wal_recovery_mode: 2 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.enable_thread_tracking: 0 2024-05-11T18:03:56.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.enable_pipelined_write: 0 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.unordered_write: 0 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.row_cache: None 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.wal_filter: None 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.allow_ingest_behind: 0 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.preserve_deletes: 0 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.two_write_queues: 0 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.manual_wal_flush: 0 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.atomic_flush: 0 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-11T18:03:56.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.log_readahead_size: 0 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.best_efforts_recovery: 0 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.allow_data_in_errors: 0 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.db_host_id: __hostname__ 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_background_jobs: 2 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_background_compactions: -1 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_subcompactions: 1 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_total_wal_size: 0 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-11T18:03:56.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_open_files: -1 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.bytes_per_sync: 0 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_readahead_size: 0 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_background_flushes: -1 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Compression algorithms supported: 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: kZSTD supported: 0 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: kXpressCompression supported: 0 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: kLZ4HCCompression supported: 1 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: kLZ4Compression supported: 1 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: kBZip2Compression supported: 0 2024-05-11T18:03:56.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: kZlibCompression supported: 1 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: kSnappyCompression supported: 1 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.merge_operator: 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_filter: None 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_filter_factory: None 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.sst_partitioner_factory: None 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557f38485d00) 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks: 1 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-11T18:03:56.992 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-11T18:03:56.993 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_top_level_index_and_filter: 1 2024-05-11T18:03:56.993 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_type: 0 2024-05-11T18:03:56.993 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_index_type: 0 2024-05-11T18:03:56.993 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_shortening: 1 2024-05-11T18:03:56.993 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-11T18:03:56.993 INFO:journalctl@ceph.mon.a.smithi044.stdout: hash_index_allow_collision: 1 2024-05-11T18:03:56.993 INFO:journalctl@ceph.mon.a.smithi044.stdout: checksum: 1 2024-05-11T18:03:56.993 INFO:journalctl@ceph.mon.a.smithi044.stdout: no_block_cache: 0 2024-05-11T18:03:56.993 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache: 0x557f384f0170 2024-05-11T18:03:56.993 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_name: BinnedLRUCache 2024-05-11T18:03:56.994 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_options: 2024-05-11T18:03:56.994 INFO:journalctl@ceph.mon.a.smithi044.stdout: capacity : 536870912 2024-05-11T18:03:56.994 INFO:journalctl@ceph.mon.a.smithi044.stdout: num_shard_bits : 4 2024-05-11T18:03:56.994 INFO:journalctl@ceph.mon.a.smithi044.stdout: strict_capacity_limit : 0 2024-05-11T18:03:56.994 INFO:journalctl@ceph.mon.a.smithi044.stdout: high_pri_pool_ratio: 0.000 2024-05-11T18:03:56.994 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_compressed: (nil) 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: persistent_cache: (nil) 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size: 4096 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size_deviation: 10 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_restart_interval: 16 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_block_restart_interval: 1 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: metadata_block_size: 4096 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: partition_filters: 0 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: use_delta_encoding: 1 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: whole_key_filtering: 1 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: verify_compression: 0 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: read_amp_bytes_per_bit: 0 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: format_version: 4 2024-05-11T18:03:56.995 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_index_compression: 1 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_align: 0 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.write_buffer_size: 33554432 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_write_buffer_number: 2 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compression: NoCompression 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.bottommost_compression: Disabled 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.prefix_extractor: nullptr 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.num_levels: 7 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-11T18:03:56.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compression_opts.level: 32767 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compression_opts.strategy: 0 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compression_opts.enabled: false 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.target_file_size_base: 67108864 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-11T18:03:56.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.arena_block_size: 4194304 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-11T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.disable_auto_compactions: 0 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.table_properties_collectors: 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.inplace_update_support: 0 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-11T18:03:56.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.bloom_locality: 0 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.max_successive_merges: 0 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.paranoid_file_checks: 0 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.force_consistency_checks: 1 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.report_bg_io_stats: 0 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.ttl: 2592000 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.enable_blob_files: false 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.min_blob_size: 0 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.blob_file_size: 268435456 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-11T18:03:57.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715450636829160, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715450636829912, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79179, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77442, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71751, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1715450636, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "4162b74a-32e1-429c-a3e9-e2e607b86f63", "db_session_id": "8IJUK0FFAFWN2F97P7GD"}} 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715450636830388, "job": 1, "event": "recovery_finished"} 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x557f384d6a80 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: DB pointer 0x557f384e6000 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** DB Stats ** 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-11T18:03:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.9 0.00 0.00 1 0.000 0 0 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.9 0.00 0.00 1 0.000 0 0 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.9 0.00 0.00 1 0.000 0 0 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-11T18:03:57.002 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.9 0.00 0.00 1 0.000 0 0 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative compaction: 0.00 GB write, 23.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval compaction: 0.00 GB write, 23.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-11T18:03:57.003 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.9 0.00 0.00 1 0.000 0 0 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.9 0.00 0.00 1 0.000 0 0 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.9 0.00 0.00 1 0.000 0 0 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-11T18:03:57.004 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative compaction: 0.00 GB write, 23.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: starting mon.a rank 0 at public addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] at bind addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: mon.a@-1(???) e1 preinit fsid ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: mon.a@-1(???).mds e1 new map 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: mon.a@-1(???).mds e1 print_map 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: e1 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: legacy client fscid: -1 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-11T18:03:57.005 INFO:journalctl@ceph.mon.a.smithi044.stdout: No filesystems configured 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta expand map: {default=false} 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta from 'false' to 'false' 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta expanded map: {default=false} 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta expand map: {default=info} 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta from 'info' to 'info' 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta expanded map: {default=info} 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta expand map: {default=daemon} 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta from 'daemon' to 'daemon' 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta expanded map: {default=daemon} 2024-05-11T18:03:57.006 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta expand map: {default=debug} 2024-05-11T18:03:57.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta from 'debug' to 'debug' 2024-05-11T18:03:57.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: expand_channel_meta expanded map: {default=debug} 2024-05-11T18:03:57.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 bash[22953]: 0c428b682c7b3044ba6f74a77637ef3aa54bb28503cd598c456a3a31e9838302 2024-05-11T18:03:57.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 systemd[1]: Started Ceph mon.a for ba7b3ba0-0fc0-11ef-bc98-c7b262605968. 2024-05-11T18:03:57.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-11T18:03:57.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: monmap e1: 1 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0]} 2024-05-11T18:03:57.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: fsmap 2024-05-11T18:03:57.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: osdmap e1: 0 total, 0 up, 0 in 2024-05-11T18:03:57.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:56 smithi044 ceph-mon[22972]: mgrmap e1: no daemons active 2024-05-11T18:03:57.933 INFO:teuthology.orchestra.run.smithi044.stderr:Wrote config to /etc/ceph/ceph.conf 2024-05-11T18:03:57.933 INFO:teuthology.orchestra.run.smithi044.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-11T18:03:57.933 INFO:teuthology.orchestra.run.smithi044.stderr:Creating mgr... 2024-05-11T18:03:57.933 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying port 9283 ... 2024-05-11T18:03:58.119 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Failed to reset failed state of unit ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mgr.y.service: Unit ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mgr.y.service not loaded. 2024-05-11T18:03:58.127 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Created symlink /etc/systemd/system/ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968.target.wants/ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@.service. 2024-05-11T18:03:58.415 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:58 smithi044 systemd[1]: Starting Ceph mgr.y for ba7b3ba0-0fc0-11ef-bc98-c7b262605968... 2024-05-11T18:03:58.775 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:58 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/3153178291' entity='client.admin' 2024-05-11T18:03:58.775 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:58 smithi044 podman[23186]: 2024-05-11 18:03:58.425528588 +0000 UTC m=+0.018840556 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-05-11T18:03:58.775 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:58 smithi044 podman[23186]: 2024-05-11 18:03:58.56602547 +0000 UTC m=+0.159337429 container create 172214ecc1951ae9321127dbbcf5be9d9698f4181a15ece30b451f409f0389d4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, release=754, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, ceph=True, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, distribution-scope=public, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , RELEASE=HEAD, name=centos-stream, io.openshift.expose-services=) 2024-05-11T18:03:58.775 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:58 smithi044 podman[23186]: 2024-05-11 18:03:58.776488074 +0000 UTC m=+0.369800036 container init 172214ecc1951ae9321127dbbcf5be9d9698f4181a15ece30b451f409f0389d4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_CLEAN=True, ceph=True, vcs-type=git, com.redhat.component=centos-stream-container, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, name=centos-stream, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, version=8, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-05-11T18:03:58.897 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: disabled 2024-05-11T18:03:58.904 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-05-11T18:03:58.905 INFO:teuthology.orchestra.run.smithi044.stderr:firewalld.service is not enabled 2024-05-11T18:03:58.905 INFO:teuthology.orchestra.run.smithi044.stderr:Not possible to enable service . firewalld.service is not available 2024-05-11T18:03:58.913 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: disabled 2024-05-11T18:03:58.921 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-05-11T18:03:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:firewalld.service is not enabled 2024-05-11T18:03:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-05-11T18:03:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mgr to start... 2024-05-11T18:03:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mgr... 2024-05-11T18:03:59.049 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:58 smithi044 podman[23186]: 2024-05-11 18:03:58.779262981 +0000 UTC m=+0.372574936 container start 172214ecc1951ae9321127dbbcf5be9d9698f4181a15ece30b451f409f0389d4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.openshift.expose-services=, vendor=Red Hat, Inc., distribution-scope=public, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, release=754, maintainer=Guillaume Abrioux , name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, version=8) 2024-05-11T18:03:59.049 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:58 smithi044 bash[23186]: 172214ecc1951ae9321127dbbcf5be9d9698f4181a15ece30b451f409f0389d4 2024-05-11T18:03:59.049 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:58 smithi044 systemd[1]: Started Ceph mgr.y for ba7b3ba0-0fc0-11ef-bc98-c7b262605968. 2024-05-11T18:03:59.049 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:58 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:03:58.926+0000 7f1e9cf32000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-11T18:03:59.399 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:59 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:03:59.068+0000 7f1e9cf32000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-11T18:03:59.399 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:59 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:03:59.173+0000 7f1e9cf32000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-11T18:03:59.402 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:03:59.402 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-05-11T18:03:59.402 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsid": "ba7b3ba0-0fc0-11ef-bc98-c7b262605968", 2024-05-11T18:03:59.402 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "health": { 2024-05-11T18:03:59.402 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-11T18:03:59.402 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "checks": {}, 2024-05-11T18:03:59.402 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mutes": [] 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum": [ 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 0 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "a" 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "monmap": { 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osdmap": { 2024-05-11T18:03:59.403 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:03:59.404 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-11T18:03:59.404 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-11T18:03:59.404 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-11T18:03:59.404 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgmap": { 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-11T18:03:59.408 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsmap": { 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "available": false, 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modules": [ 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "iostat", 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "nfs", 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "restful" 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:03:59.409 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-05-11T18:03:59.410 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:03:59.410 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "servicemap": { 2024-05-11T18:03:59.410 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:03:59.410 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modified": "2024-05-11T18:03:51.780703+0000", 2024-05-11T18:03:59.410 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-05-11T18:03:59.410 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:03:59.410 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-11T18:03:59.410 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-05-11T18:03:59.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:03:59 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/251036981' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-11T18:03:59.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:59 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:03:59.512+0000 7f1e9cf32000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-11T18:03:59.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:03:59 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:03:59.598+0000 7f1e9cf32000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-11T18:04:00.435 INFO:teuthology.orchestra.run.smithi044.stderr:mgr not available, waiting (1/15)... 2024-05-11T18:04:01.053 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:00 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:00.799+0000 7f1e9cf32000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-11T18:04:01.053 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:00 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:00.969+0000 7f1e9cf32000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-11T18:04:01.053 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:01 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:01.053+0000 7f1e9cf32000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-11T18:04:01.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:01 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:01.377+0000 7f1e9cf32000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-11T18:04:01.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:01 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:01.470+0000 7f1e9cf32000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-11T18:04:01.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:01 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:01.566+0000 7f1e9cf32000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-11T18:04:02.164 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:01 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:01.815+0000 7f1e9cf32000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-11T18:04:02.563 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:02 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:02.276+0000 7f1e9cf32000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-11T18:04:02.836 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:02 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:02.790+0000 7f1e9cf32000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-11T18:04:02.938 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsid": "ba7b3ba0-0fc0-11ef-bc98-c7b262605968", 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "health": { 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "checks": {}, 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mutes": [] 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum": [ 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 0 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "a" 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:04:02.939 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "monmap": { 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osdmap": { 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-11T18:04:02.940 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgmap": { 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsmap": { 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:02.941 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "available": false, 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modules": [ 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "iostat", 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "nfs", 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "restful" 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "servicemap": { 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modified": "2024-05-11T18:03:51.780703+0000", 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-05-11T18:04:02.942 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:02.943 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-11T18:04:02.943 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-05-11T18:04:03.165 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:02 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/375739492' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-11T18:04:03.165 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:02 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:02.889+0000 7f1e9cf32000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-11T18:04:03.573 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:03 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:03.203+0000 7f1e9cf32000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-11T18:04:03.573 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:03 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:03.296+0000 7f1e9cf32000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-11T18:04:03.576 INFO:teuthology.orchestra.run.smithi044.stderr:mgr not available, waiting (2/15)... 2024-05-11T18:04:03.851 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:03 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:03.714+0000 7f1e9cf32000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-11T18:04:03.851 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:03 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:03.851+0000 7f1e9cf32000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-11T18:04:04.155 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:03 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:03.946+0000 7f1e9cf32000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-11T18:04:04.155 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:04 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:04.063+0000 7f1e9cf32000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-11T18:04:04.155 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:04 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:04.155+0000 7f1e9cf32000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-11T18:04:05.096 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:04 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:04.724+0000 7f1e9cf32000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-11T18:04:05.415 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:05 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:05.097+0000 7f1e9cf32000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-11T18:04:05.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: Activating manager daemon y 2024-05-11T18:04:05.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: mgrmap e2: y(active, starting, since 0.00268634s) 2024-05-11T18:04:05.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: from='mgr.14100 172.21.15.44:0/565279156' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-11T18:04:05.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: from='mgr.14100 172.21.15.44:0/565279156' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-11T18:04:05.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: from='mgr.14100 172.21.15.44:0/565279156' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-11T18:04:05.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: from='mgr.14100 172.21.15.44:0/565279156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-11T18:04:05.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: from='mgr.14100 172.21.15.44:0/565279156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-11T18:04:05.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: Manager daemon y is now available 2024-05-11T18:04:05.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: from='mgr.14100 172.21.15.44:0/565279156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-11T18:04:05.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: from='mgr.14100 172.21.15.44:0/565279156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-11T18:04:05.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: from='mgr.14100 172.21.15.44:0/565279156' entity='mgr.y' 2024-05-11T18:04:05.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: from='mgr.14100 172.21.15.44:0/565279156' entity='mgr.y' 2024-05-11T18:04:05.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:05 smithi044 ceph-mon[22972]: from='mgr.14100 172.21.15.44:0/565279156' entity='mgr.y' 2024-05-11T18:04:06.057 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:04:06.057 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsid": "ba7b3ba0-0fc0-11ef-bc98-c7b262605968", 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "health": { 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "checks": {}, 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mutes": [] 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum": [ 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 0 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "a" 2024-05-11T18:04:06.058 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "monmap": { 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osdmap": { 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:06.059 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgmap": { 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsmap": { 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-11T18:04:06.060 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "available": false, 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modules": [ 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "iostat", 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "nfs", 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "restful" 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "servicemap": { 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modified": "2024-05-11T18:03:51.780703+0000", 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-11T18:04:06.061 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-05-11T18:04:06.414 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:06 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/1233855655' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-11T18:04:06.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:06 smithi044 ceph-mon[22972]: mgrmap e3: y(active, since 1.00528s) 2024-05-11T18:04:06.582 INFO:teuthology.orchestra.run.smithi044.stderr:mgr not available, waiting (3/15)... 2024-05-11T18:04:08.965 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:08 smithi044 ceph-mon[22972]: mgrmap e4: y(active, since 2s) 2024-05-11T18:04:09.334 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsid": "ba7b3ba0-0fc0-11ef-bc98-c7b262605968", 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "health": { 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "checks": {}, 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mutes": [] 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum": [ 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 0 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:04:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "a" 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "monmap": { 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osdmap": { 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-11T18:04:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgmap": { 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-11T18:04:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsmap": { 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "available": true, 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modules": [ 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "iostat", 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "nfs", 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "restful" 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-05-11T18:04:09.338 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-05-11T18:04:09.339 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:09.339 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "servicemap": { 2024-05-11T18:04:09.339 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-11T18:04:09.339 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modified": "2024-05-11T18:03:51.780703+0000", 2024-05-11T18:04:09.339 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-05-11T18:04:09.339 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-05-11T18:04:09.339 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-11T18:04:09.339 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-05-11T18:04:09.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:09 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/4103475179' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-11T18:04:09.954 INFO:teuthology.orchestra.run.smithi044.stderr:mgr is available 2024-05-11T18:04:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:04:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: [global] 2024-05-11T18:04:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: fsid = ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:04:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-05-11T18:04:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-05-11T18:04:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-05-11T18:04:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-05-11T18:04:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:04:10.540 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: [mgr] 2024-05-11T18:04:10.541 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-05-11T18:04:10.541 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-05-11T18:04:10.541 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: [osd] 2024-05-11T18:04:10.541 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-05-11T18:04:10.541 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-05-11T18:04:11.108 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:10 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/1630266039' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-11T18:04:11.109 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:10 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/1630266039' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-11T18:04:11.111 INFO:teuthology.orchestra.run.smithi044.stderr:Enabling cephadm module... 2024-05-11T18:04:12.142 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:11 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/3675412550' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-11T18:04:12.142 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:11 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: ignoring --setuser ceph since I am not root 2024-05-11T18:04:12.142 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:11 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: ignoring --setgroup ceph since I am not root 2024-05-11T18:04:12.142 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:12 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:12.006+0000 7f1bce156000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-11T18:04:12.142 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:12 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:12.141+0000 7f1bce156000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-11T18:04:12.414 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:12 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:12.243+0000 7f1bce156000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-11T18:04:12.673 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:12 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:12.565+0000 7f1bce156000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-11T18:04:12.673 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:12 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:12.651+0000 7f1bce156000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-11T18:04:13.165 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:12 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/3675412550' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-11T18:04:13.165 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:12 smithi044 ceph-mon[22972]: mgrmap e5: y(active, since 6s) 2024-05-11T18:04:13.169 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-05-11T18:04:13.169 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 5, 2024-05-11T18:04:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "available": true, 2024-05-11T18:04:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "active_name": "y", 2024-05-11T18:04:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_standby": 0 2024-05-11T18:04:13.170 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-05-11T18:04:13.774 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for the mgr to restart... 2024-05-11T18:04:13.774 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mgr epoch 5... 2024-05-11T18:04:14.034 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:13 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/2437955117' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-11T18:04:14.034 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:13 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:13.860+0000 7f1bce156000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-11T18:04:14.034 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:14 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:14.030+0000 7f1bce156000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-11T18:04:14.291 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:14 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:14.117+0000 7f1bce156000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-11T18:04:14.634 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:14 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:14.445+0000 7f1bce156000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-11T18:04:14.634 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:14 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:14.538+0000 7f1bce156000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-11T18:04:14.635 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:14 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:14.633+0000 7f1bce156000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-11T18:04:14.914 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:14 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:14.882+0000 7f1bce156000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-11T18:04:15.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:15 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:15.340+0000 7f1bce156000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-11T18:04:16.164 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:15 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:15.843+0000 7f1bce156000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-11T18:04:16.165 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:15 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:15.939+0000 7f1bce156000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-11T18:04:16.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:16 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:16.243+0000 7f1bce156000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-11T18:04:16.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:16 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:16.335+0000 7f1bce156000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-11T18:04:17.100 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:16 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:16.751+0000 7f1bce156000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-11T18:04:17.100 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:16 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:16.888+0000 7f1bce156000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-11T18:04:17.100 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:16 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:16.982+0000 7f1bce156000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-11T18:04:17.100 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:17 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:17.099+0000 7f1bce156000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-11T18:04:17.414 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:17 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:17.190+0000 7f1bce156000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-11T18:04:18.131 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:17 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:17.756+0000 7f1bce156000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-11T18:04:18.414 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:18 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:18.131+0000 7f1bce156000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-11T18:04:18.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:18 smithi044 ceph-mon[22972]: Active manager daemon y restarted 2024-05-11T18:04:18.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:18 smithi044 ceph-mon[22972]: Activating manager daemon y 2024-05-11T18:04:18.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:18 smithi044 ceph-mon[22972]: osdmap e2: 0 total, 0 up, 0 in 2024-05-11T18:04:19.199 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-05-11T18:04:19.199 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-05-11T18:04:19.199 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "initialized": true 2024-05-11T18:04:19.199 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-05-11T18:04:19.586 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: mgrmap e6: y(active, starting, since 0.0526347s) 2024-05-11T18:04:19.586 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-11T18:04:19.586 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-11T18:04:19.586 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-11T18:04:19.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-11T18:04:19.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-11T18:04:19.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: Manager daemon y is now available 2024-05-11T18:04:19.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:19.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:19.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-11T18:04:19.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-11T18:04:19.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-11T18:04:19.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-11T18:04:19.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:19 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-11T18:04:19.888 INFO:teuthology.orchestra.run.smithi044.stderr:mgr epoch 5 is available 2024-05-11T18:04:19.888 INFO:teuthology.orchestra.run.smithi044.stderr:Setting orchestrator backend to cephadm... 2024-05-11T18:04:20.138 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:20 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: [11/May/2024:18:04:20] ENGINE Bus STARTING 2024-05-11T18:04:20.138 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:20 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: [11/May/2024:18:04:20] ENGINE Serving on https://172.21.15.44:7150 2024-05-11T18:04:20.415 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:20 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: [11/May/2024:18:04:20] ENGINE Bus STARTED 2024-05-11T18:04:20.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:20 smithi044 ceph-mon[22972]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-11T18:04:20.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:20 smithi044 ceph-mon[22972]: mgrmap e7: y(active, since 1.05965s) 2024-05-11T18:04:20.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:20 smithi044 ceph-mon[22972]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-11T18:04:20.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:20 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:20.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:20 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-11T18:04:21.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:21 smithi044 ceph-mon[22972]: [11/May/2024:18:04:20] ENGINE Bus STARTING 2024-05-11T18:04:21.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:21 smithi044 ceph-mon[22972]: [11/May/2024:18:04:20] ENGINE Serving on https://172.21.15.44:7150 2024-05-11T18:04:21.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:21 smithi044 ceph-mon[22972]: [11/May/2024:18:04:20] ENGINE Bus STARTED 2024-05-11T18:04:21.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:21 smithi044 ceph-mon[22972]: mgrmap e8: y(active, since 2s) 2024-05-11T18:04:21.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:21 smithi044 ceph-mon[22972]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-11T18:04:21.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:21 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:21.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:21 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-11T18:04:21.728 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: value unchanged 2024-05-11T18:04:22.310 INFO:teuthology.orchestra.run.smithi044.stderr:Generating ssh key... 2024-05-11T18:04:22.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:22 smithi044 ceph-mon[22972]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-11T18:04:23.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: Generating public/private rsa key pair. 2024-05-11T18:04:23.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: Your identification has been saved in /tmp/tmprlb9is5k/key. 2024-05-11T18:04:23.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: Your public key has been saved in /tmp/tmprlb9is5k/key.pub. 2024-05-11T18:04:23.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: The key fingerprint is: 2024-05-11T18:04:23.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: SHA256:dyuYArqegdgCG7Owu/PXLaCw+S5tP/Yla9fNEg+tv+0 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:04:23.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: The key's randomart image is: 2024-05-11T18:04:23.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: +---[RSA 3072]----+ 2024-05-11T18:04:23.666 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: | | 2024-05-11T18:04:23.666 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: | | 2024-05-11T18:04:23.666 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: | | 2024-05-11T18:04:23.666 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: | | 2024-05-11T18:04:23.666 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: |= . S ... | 2024-05-11T18:04:23.666 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: |=O ... +o... | 2024-05-11T18:04:23.666 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: |B=+. oo.+..B. | 2024-05-11T18:04:23.666 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: |=o+++ +=o +.+. | 2024-05-11T18:04:23.666 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: |oXB+.+oo oooE | 2024-05-11T18:04:23.666 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:23 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: +----[SHA256]-----+ 2024-05-11T18:04:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:24 smithi044 ceph-mon[22972]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-11T18:04:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:24 smithi044 ceph-mon[22972]: Generating ssh key... 2024-05-11T18:04:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:24 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:24 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:24.516 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1JEZCpq4H1yqoo8tKW3A7NEiQ60uxUkRORhWZ/honCK1jawKQpdCG9kwnX2hSvW7Iej1e53ZaRftqEHi7EGeWJTCfdNT30mPeSgUj+2NQRCbfNYn1BfqXJVvpXUIlX8LNsfzTuoiTKlPxlAKhElpVIkBmNIX3R9g7xDwI0LqJ2cdz2wj0s+uv3t9y4rnYVGoNT3+YoIigrzjh8rDY0F5GBL958NZRd6Od0VP8m5YfT+v3knvznf2kGcezvxOSOeWjt6MLDLFXliWeGLVSlNE8lyprXfFYf7t/qE+7MM4AYcPL4KemDMcNjcLiAjjaInJBxJUteIBLVDxFb6G2Qe/VvBQbWWw8CNI1unhaPMf8jAr4XUzme9I7GRdn6J/Z6VKZYye7KeccQxdahTojfQr74iUpUadTGbrQZoENXuFViF43ynIXzmcFeWhDyW31zrH71zDAUZnD+WQ3VsAIzrW1OD6xyqr4bEiG0bzzJutrz1A6IMVh4vqFDi3nLlqf178= ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:04:25.073 INFO:teuthology.orchestra.run.smithi044.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-11T18:04:25.073 INFO:teuthology.orchestra.run.smithi044.stderr:Adding key to root@localhost authorized_keys... 2024-05-11T18:04:25.074 INFO:teuthology.orchestra.run.smithi044.stderr:Adding host smithi044... 2024-05-11T18:04:25.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:25 smithi044 ceph-mon[22972]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-11T18:04:26.634 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: Added host 'smithi044' with addr '172.21.15.44' 2024-05-11T18:04:26.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:26 smithi044 ceph-mon[22972]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi044", "addr": "172.21.15.44", "target": ["mon-mgr", ""]}]: dispatch 2024-05-11T18:04:27.464 INFO:teuthology.orchestra.run.smithi044.stderr:Deploying unmanaged mon service... 2024-05-11T18:04:27.742 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:27 smithi044 ceph-mon[22972]: Deploying cephadm binary to smithi044 2024-05-11T18:04:27.743 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:27 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:27.743 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:27 smithi044 ceph-mon[22972]: Added host smithi044 2024-05-11T18:04:27.743 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:27 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-11T18:04:28.230 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: Scheduled mon update... 2024-05-11T18:04:29.050 INFO:teuthology.orchestra.run.smithi044.stderr:Deploying unmanaged mgr service... 2024-05-11T18:04:29.353 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:29 smithi044 ceph-mon[22972]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-11T18:04:29.353 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:29 smithi044 ceph-mon[22972]: Saving service mon spec with placement count:5 2024-05-11T18:04:29.353 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:29 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:29.880 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-05-11T18:04:31.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:30 smithi044 ceph-mon[22972]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-11T18:04:31.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:30 smithi044 ceph-mon[22972]: Saving service mgr spec with placement count:2 2024-05-11T18:04:31.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:30 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:31.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:30 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:31.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:30 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:32.638 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:32 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/4284984399' entity='client.admin' 2024-05-11T18:04:33.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:33 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:33.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:33 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/3515049342' entity='client.admin' 2024-05-11T18:04:33.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:33 smithi044 ceph-mon[22972]: from='mgr.14120 172.21.15.44:0/1422484630' entity='mgr.y' 2024-05-11T18:04:33.910 INFO:teuthology.orchestra.run.smithi044.stderr:Enabling the dashboard module... 2024-05-11T18:04:34.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:34 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/135553854' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-11T18:04:34.883 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:34 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: ignoring --setuser ceph since I am not root 2024-05-11T18:04:34.883 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:34 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: ignoring --setgroup ceph since I am not root 2024-05-11T18:04:34.883 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:34 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:34.882+0000 7f0d62fe2000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-11T18:04:35.164 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:35 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:35.016+0000 7f0d62fe2000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-11T18:04:35.165 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:35 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:35.120+0000 7f0d62fe2000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-11T18:04:35.719 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:35 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:35.439+0000 7f0d62fe2000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-11T18:04:35.719 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:35 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:35.524+0000 7f0d62fe2000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-11T18:04:36.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:35 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/135553854' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-11T18:04:36.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:35 smithi044 ceph-mon[22972]: mgrmap e9: y(active, since 16s) 2024-05-11T18:04:36.881 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:36 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:36.706+0000 7f0d62fe2000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-11T18:04:36.881 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:36 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:36.881+0000 7f0d62fe2000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-11T18:04:37.067 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-05-11T18:04:37.067 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 9, 2024-05-11T18:04:37.067 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "available": true, 2024-05-11T18:04:37.067 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "active_name": "y", 2024-05-11T18:04:37.067 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_standby": 0 2024-05-11T18:04:37.067 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-05-11T18:04:37.164 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:36 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:36.966+0000 7f0d62fe2000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-11T18:04:37.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:37 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:37.294+0000 7f0d62fe2000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-11T18:04:37.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:37 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:37.390+0000 7f0d62fe2000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-11T18:04:37.665 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:37 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:37.487+0000 7f0d62fe2000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-11T18:04:37.771 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for the mgr to restart... 2024-05-11T18:04:37.771 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mgr epoch 9... 2024-05-11T18:04:37.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:37 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/2230783267' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-11T18:04:37.989 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:37 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:37.736+0000 7f0d62fe2000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-11T18:04:38.415 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:38 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:38.208+0000 7f0d62fe2000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-11T18:04:39.112 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:38 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:38.711+0000 7f0d62fe2000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-11T18:04:39.112 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:38 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:38.807+0000 7f0d62fe2000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-11T18:04:39.113 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:39 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:39.112+0000 7f0d62fe2000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-11T18:04:39.415 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:39 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:39.204+0000 7f0d62fe2000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-11T18:04:39.767 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:39 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:39.625+0000 7f0d62fe2000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-11T18:04:39.768 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:39 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:39.767+0000 7f0d62fe2000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-11T18:04:40.081 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:39 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:39.866+0000 7f0d62fe2000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-11T18:04:40.081 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:39 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:39.986+0000 7f0d62fe2000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-11T18:04:40.082 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:40 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:40.081+0000 7f0d62fe2000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-11T18:04:40.915 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:40 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:40.654+0000 7f0d62fe2000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-11T18:04:41.301 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:41 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: 2024-05-11T18:04:41.027+0000 7f0d62fe2000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-11T18:04:41.301 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:41 smithi044 ceph-mon[22972]: Active manager daemon y restarted 2024-05-11T18:04:41.302 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:41 smithi044 ceph-mon[22972]: Activating manager daemon y 2024-05-11T18:04:41.302 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:41 smithi044 ceph-mon[22972]: osdmap e3: 0 total, 0 up, 0 in 2024-05-11T18:04:41.568 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:41 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: [11/May/2024:18:04:41] ENGINE Bus STARTING 2024-05-11T18:04:41.824 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:41 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: [11/May/2024:18:04:41] ENGINE Serving on https://172.21.15.44:7150 2024-05-11T18:04:41.824 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:41 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y[23200]: [11/May/2024:18:04:41] ENGINE Bus STARTED 2024-05-11T18:04:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-05-11T18:04:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-05-11T18:04:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "initialized": true 2024-05-11T18:04:42.100 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-05-11T18:04:42.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: mgrmap e10: y(active, starting, since 0.0554845s) 2024-05-11T18:04:42.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-11T18:04:42.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-11T18:04:42.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-11T18:04:42.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-11T18:04:42.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-11T18:04:42.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: Manager daemon y is now available 2024-05-11T18:04:42.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-11T18:04:42.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-11T18:04:42.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-11T18:04:42.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-11T18:04:42.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: [11/May/2024:18:04:41] ENGINE Bus STARTING 2024-05-11T18:04:42.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: [11/May/2024:18:04:41] ENGINE Serving on https://172.21.15.44:7150 2024-05-11T18:04:42.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: [11/May/2024:18:04:41] ENGINE Bus STARTED 2024-05-11T18:04:42.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:42 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' 2024-05-11T18:04:42.795 INFO:teuthology.orchestra.run.smithi044.stderr:mgr epoch 9 is available 2024-05-11T18:04:42.795 INFO:teuthology.orchestra.run.smithi044.stderr:Generating a dashboard self-signed certificate... 2024-05-11T18:04:43.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:43 smithi044 ceph-mon[22972]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-11T18:04:43.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:43 smithi044 ceph-mon[22972]: mgrmap e11: y(active, since 1.06318s) 2024-05-11T18:04:43.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:43 smithi044 ceph-mon[22972]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-11T18:04:43.531 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: Self-signed certificate created 2024-05-11T18:04:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:Creating initial admin user... 2024-05-11T18:04:44.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:44 smithi044 ceph-mon[22972]: mgrmap e12: y(active, since 2s) 2024-05-11T18:04:44.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:44 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' 2024-05-11T18:04:44.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:44 smithi044 ceph-mon[22972]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-11T18:04:44.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:44 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' 2024-05-11T18:04:44.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:44 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' 2024-05-11T18:04:45.098 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$SRHMMzBWz4DDLO9vfjnF4uR9Iy.d3k1mToKGnmd.mucnBAT7tpiDy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715450685, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-11T18:04:45.414 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:45 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' 2024-05-11T18:04:45.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:45 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' 2024-05-11T18:04:45.880 INFO:teuthology.orchestra.run.smithi044.stderr:Fetching dashboard port number... 2024-05-11T18:04:46.467 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:46 smithi044 ceph-mon[22972]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-11T18:04:46.957 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 8443 2024-05-11T18:04:47.663 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: disabled 2024-05-11T18:04:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-05-11T18:04:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:firewalld.service is not enabled 2024-05-11T18:04:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-11T18:04:47.675 INFO:teuthology.orchestra.run.smithi044.stderr:Ceph Dashboard is now available at: 2024-05-11T18:04:47.675 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-05-11T18:04:47.675 INFO:teuthology.orchestra.run.smithi044.stderr: URL: https://smithi044.front.sepia.ceph.com:8443/ 2024-05-11T18:04:47.675 INFO:teuthology.orchestra.run.smithi044.stderr: User: admin 2024-05-11T18:04:47.675 INFO:teuthology.orchestra.run.smithi044.stderr: Password: 8m5bhxpx2r 2024-05-11T18:04:47.675 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-05-11T18:04:47.676 INFO:teuthology.orchestra.run.smithi044.stderr:Enabling autotune for osd_memory_target 2024-05-11T18:04:47.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:47 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/3367558726' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-11T18:04:49.100 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:48 smithi044 ceph-mon[22972]: mgrmap e13: y(active, since 6s) 2024-05-11T18:04:49.822 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-05-11T18:04:50.392 INFO:teuthology.orchestra.run.smithi044.stderr:You can access the Ceph CLI with: 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid ba7b3ba0-0fc0-11ef-bc98-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:Please consider enabling telemetry to help improve Ceph: 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr: ceph telemetry on 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:For more information see: 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-05-11T18:04:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:Bootstrap complete. 2024-05-11T18:04:50.427 INFO:tasks.cephadm:Fetching config... 2024-05-11T18:04:50.427 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-11T18:04:50.427 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-11T18:04:50.444 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-11T18:04:50.444 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-11T18:04:50.444 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-11T18:04:50.500 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-11T18:04:50.501 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-11T18:04:50.501 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-11T18:04:50.567 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-11T18:04:50.567 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-11T18:04:50.567 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-11T18:04:50.623 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-11T18:04:50.624 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1JEZCpq4H1yqoo8tKW3A7NEiQ60uxUkRORhWZ/honCK1jawKQpdCG9kwnX2hSvW7Iej1e53ZaRftqEHi7EGeWJTCfdNT30mPeSgUj+2NQRCbfNYn1BfqXJVvpXUIlX8LNsfzTuoiTKlPxlAKhElpVIkBmNIX3R9g7xDwI0LqJ2cdz2wj0s+uv3t9y4rnYVGoNT3+YoIigrzjh8rDY0F5GBL958NZRd6Od0VP8m5YfT+v3knvznf2kGcezvxOSOeWjt6MLDLFXliWeGLVSlNE8lyprXfFYf7t/qE+7MM4AYcPL4KemDMcNjcLiAjjaInJBxJUteIBLVDxFb6G2Qe/VvBQbWWw8CNI1unhaPMf8jAr4XUzme9I7GRdn6J/Z6VKZYye7KeccQxdahTojfQr74iUpUadTGbrQZoENXuFViF43ynIXzmcFeWhDyW31zrH71zDAUZnD+WQ3VsAIzrW1OD6xyqr4bEiG0bzzJutrz1A6IMVh4vqFDi3nLlqf178= ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-11T18:04:50.699 INFO:teuthology.orchestra.run.smithi044.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1JEZCpq4H1yqoo8tKW3A7NEiQ60uxUkRORhWZ/honCK1jawKQpdCG9kwnX2hSvW7Iej1e53ZaRftqEHi7EGeWJTCfdNT30mPeSgUj+2NQRCbfNYn1BfqXJVvpXUIlX8LNsfzTuoiTKlPxlAKhElpVIkBmNIX3R9g7xDwI0LqJ2cdz2wj0s+uv3t9y4rnYVGoNT3+YoIigrzjh8rDY0F5GBL958NZRd6Od0VP8m5YfT+v3knvznf2kGcezvxOSOeWjt6MLDLFXliWeGLVSlNE8lyprXfFYf7t/qE+7MM4AYcPL4KemDMcNjcLiAjjaInJBxJUteIBLVDxFb6G2Qe/VvBQbWWw8CNI1unhaPMf8jAr4XUzme9I7GRdn6J/Z6VKZYye7KeccQxdahTojfQr74iUpUadTGbrQZoENXuFViF43ynIXzmcFeWhDyW31zrH71zDAUZnD+WQ3VsAIzrW1OD6xyqr4bEiG0bzzJutrz1A6IMVh4vqFDi3nLlqf178= ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:04:50.712 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1JEZCpq4H1yqoo8tKW3A7NEiQ60uxUkRORhWZ/honCK1jawKQpdCG9kwnX2hSvW7Iej1e53ZaRftqEHi7EGeWJTCfdNT30mPeSgUj+2NQRCbfNYn1BfqXJVvpXUIlX8LNsfzTuoiTKlPxlAKhElpVIkBmNIX3R9g7xDwI0LqJ2cdz2wj0s+uv3t9y4rnYVGoNT3+YoIigrzjh8rDY0F5GBL958NZRd6Od0VP8m5YfT+v3knvznf2kGcezvxOSOeWjt6MLDLFXliWeGLVSlNE8lyprXfFYf7t/qE+7MM4AYcPL4KemDMcNjcLiAjjaInJBxJUteIBLVDxFb6G2Qe/VvBQbWWw8CNI1unhaPMf8jAr4XUzme9I7GRdn6J/Z6VKZYye7KeccQxdahTojfQr74iUpUadTGbrQZoENXuFViF43ynIXzmcFeWhDyW31zrH71zDAUZnD+WQ3VsAIzrW1OD6xyqr4bEiG0bzzJutrz1A6IMVh4vqFDi3nLlqf178= ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-11T18:04:50.751 INFO:teuthology.orchestra.run.smithi092.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1JEZCpq4H1yqoo8tKW3A7NEiQ60uxUkRORhWZ/honCK1jawKQpdCG9kwnX2hSvW7Iej1e53ZaRftqEHi7EGeWJTCfdNT30mPeSgUj+2NQRCbfNYn1BfqXJVvpXUIlX8LNsfzTuoiTKlPxlAKhElpVIkBmNIX3R9g7xDwI0LqJ2cdz2wj0s+uv3t9y4rnYVGoNT3+YoIigrzjh8rDY0F5GBL958NZRd6Od0VP8m5YfT+v3knvznf2kGcezvxOSOeWjt6MLDLFXliWeGLVSlNE8lyprXfFYf7t/qE+7MM4AYcPL4KemDMcNjcLiAjjaInJBxJUteIBLVDxFb6G2Qe/VvBQbWWw8CNI1unhaPMf8jAr4XUzme9I7GRdn6J/Z6VKZYye7KeccQxdahTojfQr74iUpUadTGbrQZoENXuFViF43ynIXzmcFeWhDyW31zrH71zDAUZnD+WQ3VsAIzrW1OD6xyqr4bEiG0bzzJutrz1A6IMVh4vqFDi3nLlqf178= ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968 2024-05-11T18:04:50.763 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ba7b3ba0-0fc0-11ef-bc98-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-11T18:04:51.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:50 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/2790975807' entity='client.admin' 2024-05-11T18:04:53.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:53 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' 2024-05-11T18:04:53.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:53 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-05-11T18:04:53.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:53 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' 2024-05-11T18:04:53.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:53 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-11T18:04:53.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:53 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' 2024-05-11T18:04:54.031 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-11T18:04:54.031 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ba7b3ba0-0fc0-11ef-bc98-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-11T18:04:54.491 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:54 smithi044 ceph-mon[22972]: from='client.? 172.21.15.44:0/195547748' entity='client.admin' 2024-05-11T18:04:54.491 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:54 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-11T18:04:54.491 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:54 smithi044 ceph-mon[22972]: from='mgr.14152 172.21.15.44:0/187211238' entity='mgr.y' 2024-05-11T18:04:57.383 DEBUG:teuthology.orchestra.run.smithi044:> echo '# created by cephadm 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> /var/log/ceph/{{ fsid }}/*.log { 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> rotate 100 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> size 10G 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> compress 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> sharedscripts 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> postrotate 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> killall -q -1 {{ targets|join('"'"' '"'"') }} || pkill -1 -x '"'"'{{ targets|join('"'"'|'"'"') }}'"'"' || true 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> endscript 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> missingok 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> notifempty 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> su root root 2024-05-11T18:04:57.384 DEBUG:teuthology.orchestra.run.smithi044:> }' > /root/logrotate.conf 2024-05-11T18:04:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-11T18:04:57.400 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 13: /root/logrotate.conf: Permission denied 2024-05-11T18:04:57.400 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-11T18:04:57.400 DEBUG:teuthology.orchestra.run.smithi044:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-11T18:04:57.456 DEBUG:teuthology.orchestra.run.smithi092:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-11T18:04:57.474 INFO:tasks.cephadm:Stopping all daemons... 2024-05-11T18:04:57.475 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-11T18:04:57.475 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mon.a 2024-05-11T18:04:57.664 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:57 smithi044 systemd[1]: Stopping Ceph mon.a for ba7b3ba0-0fc0-11ef-bc98-c7b262605968... 2024-05-11T18:04:57.664 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:57 smithi044 bash[27089]: Error: no container with name or ID "ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon.a" found: no such container 2024-05-11T18:04:57.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:57 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a[22967]: 2024-05-11T18:04:57.664+0000 7f5fbcd59700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-11T18:04:57.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:57 smithi044 ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a[22967]: 2024-05-11T18:04:57.664+0000 7f5fbcd59700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-11T18:04:57.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:57 smithi044 podman[27100]: 2024-05-11 18:04:57.73641476 +0000 UTC m=+0.161876570 container died 0c428b682c7b3044ba6f74a77637ef3aa54bb28503cd598c456a3a31e9838302 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, RELEASE=HEAD, version=8, ceph=True, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-05-11T18:04:58.236 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:58 smithi044 podman[27100]: 2024-05-11 18:04:58.05532007 +0000 UTC m=+0.480781887 container cleanup 0c428b682c7b3044ba6f74a77637ef3aa54bb28503cd598c456a3a31e9838302 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, ceph=True, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, distribution-scope=public, io.buildah.version=1.19.8, name=centos-stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, version=8) 2024-05-11T18:04:58.236 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:58 smithi044 bash[27100]: ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a 2024-05-11T18:04:58.236 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 11 18:04:58 smithi044 podman[27114]: 2024-05-11 18:04:58.215012414 +0000 UTC m=+0.476124186 container remove 0c428b682c7b3044ba6f74a77637ef3aa54bb28503cd598c456a3a31e9838302 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mon-a, RELEASE=HEAD, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., name=centos-stream, GIT_CLEAN=True, architecture=x86_64, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, distribution-scope=public, release=754, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8) 2024-05-11T18:04:58.254 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mon.a.service' 2024-05-11T18:04:58.319 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-11T18:04:58.319 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-11T18:04:58.319 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-11T18:04:58.320 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mgr.y 2024-05-11T18:04:58.530 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:58 smithi044 systemd[1]: Stopping Ceph mgr.y for ba7b3ba0-0fc0-11ef-bc98-c7b262605968... 2024-05-11T18:04:58.530 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:58 smithi044 bash[27179]: Error: no container with name or ID "ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr.y" found: no such container 2024-05-11T18:04:58.817 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:58 smithi044 podman[27190]: 2024-05-11 18:04:58.531071775 +0000 UTC m=+0.134088810 container died 172214ecc1951ae9321127dbbcf5be9d9698f4181a15ece30b451f409f0389d4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y, GIT_CLEAN=True, distribution-scope=public, version=8, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., RELEASE=HEAD, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-05-11T18:04:59.151 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:58 smithi044 podman[27190]: 2024-05-11 18:04:58.893006245 +0000 UTC m=+0.496023270 container cleanup 172214ecc1951ae9321127dbbcf5be9d9698f4181a15ece30b451f409f0389d4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, vcs-type=git, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, distribution-scope=public, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, release=754, version=8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-11T18:04:59.151 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:58 smithi044 bash[27190]: ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr-y 2024-05-11T18:04:59.152 INFO:journalctl@ceph.mgr.y.smithi044.stdout:May 11 18:04:58 smithi044 bash[27217]: Error: no container with name or ID "ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968-mgr.y" found: no such container 2024-05-11T18:04:59.158 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ba7b3ba0-0fc0-11ef-bc98-c7b262605968@mgr.y.service' 2024-05-11T18:04:59.196 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-11T18:04:59.196 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-11T18:04:59.196 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ba7b3ba0-0fc0-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-11T18:05:00.100 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ba7b3ba0-0fc0-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-11T18:05:00.372 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-11T18:05:00.400 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-11T18:05:00.425 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d/qa/tasks/cephadm.py", line 838, in ceph_bootstrap bootstrap_remote.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 1: 'echo \'# created by cephadm\n/var/log/ceph/{{ fsid }}/*.log {\n rotate 100\n size 10G\n compress\n sharedscripts\n postrotate\n killall -q -1 {{ targets|join(\'"\'"\' \'"\'"\') }} || pkill -1 -x \'"\'"\'{{ targets|join(\'"\'"\'|\'"\'"\') }}\'"\'"\' || true\n endscript\n missingok\n notifempty\n su root root\n}\' > /root/logrotate.conf' 2024-05-11T18:05:00.427 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-11T18:05:00.429 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525/remote/smithi044/crash 2024-05-11T18:05:00.430 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/lib/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/crash -- . 2024-05-11T18:05:00.482 INFO:teuthology.orchestra.run.smithi044.stderr:tar: /var/lib/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-11T18:05:00.482 INFO:teuthology.orchestra.run.smithi044.stderr:tar: Error is not recoverable: exiting now 2024-05-11T18:05:00.484 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525/remote/smithi092/crash 2024-05-11T18:05:00.485 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/crash -- . 2024-05-11T18:05:00.535 INFO:teuthology.orchestra.run.smithi092.stderr:tar: /var/lib/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-11T18:05:00.535 INFO:teuthology.orchestra.run.smithi092.stderr:tar: Error is not recoverable: exiting now 2024-05-11T18:05:00.537 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-11T18:05:00.537 DEBUG:teuthology.orchestra.run.smithi044:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | head -n 1 2024-05-11T18:05:00.565 INFO:tasks.cephadm:Compressing logs... 2024-05-11T18:05:00.565 DEBUG:teuthology.orchestra.run.smithi044:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-11T18:05:00.608 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-11T18:05:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-11T18:05:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:: No such file or directory 2024-05-11T18:05:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph.log 2024-05-11T18:05:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-11T18:05:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph-mgr.y.log 2024-05-11T18:05:00.633 INFO:teuthology.orchestra.run.smithi044.stderr: 79.2% -- replaced with /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-11T18:05:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph-mon.a.log 2024-05-11T18:05:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph-mgr.y.log: 86.4% -- replaced with /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-11T18:05:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-11T18:05:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-11T18:05:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph.cephadm.log: 71.2% -- replaced with /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-11T18:05:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph-volume.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-11T18:05:00.634 INFO:teuthology.orchestra.run.smithi044.stderr: 91.1% -- replaced with /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-11T18:05:00.635 INFO:teuthology.orchestra.run.smithi092.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-11T18:05:00.635 INFO:teuthology.orchestra.run.smithi092.stderr:: No such file or directory 2024-05-11T18:05:00.635 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/cephadm.log: 69.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-11T18:05:00.635 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-05-11T18:05:00.635 INFO:teuthology.orchestra.run.smithi092.stderr:real 0m0.012s 2024-05-11T18:05:00.635 INFO:teuthology.orchestra.run.smithi092.stderr:user 0m0.011s 2024-05-11T18:05:00.635 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m0.010s 2024-05-11T18:05:00.650 INFO:teuthology.orchestra.run.smithi044.stderr: 88.4% -- replaced with /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph-mgr.y.log.gz 2024-05-11T18:05:00.656 INFO:teuthology.orchestra.run.smithi044.stderr: 91.2% -- replaced with /var/log/ceph/ba7b3ba0-0fc0-11ef-bc98-c7b262605968/ceph-mon.a.log.gz 2024-05-11T18:05:00.658 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-05-11T18:05:00.658 INFO:teuthology.orchestra.run.smithi044.stderr:real 0m0.038s 2024-05-11T18:05:00.658 INFO:teuthology.orchestra.run.smithi044.stderr:user 0m0.059s 2024-05-11T18:05:00.659 INFO:teuthology.orchestra.run.smithi044.stderr:sys 0m0.014s 2024-05-11T18:05:00.659 INFO:tasks.cephadm:Archiving logs... 2024-05-11T18:05:00.659 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/log/ceph to /home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525/remote/smithi044/log 2024-05-11T18:05:00.660 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-11T18:05:00.737 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525/remote/smithi092/log 2024-05-11T18:05:00.737 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-11T18:05:00.762 INFO:tasks.cephadm:Removing cluster... 2024-05-11T18:05:00.763 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ba7b3ba0-0fc0-11ef-bc98-c7b262605968 --force 2024-05-11T18:05:01.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ba7b3ba0-0fc0-11ef-bc98-c7b262605968 --force 2024-05-11T18:05:01.315 INFO:tasks.cephadm:Removing cephadm ... 2024-05-11T18:05:01.315 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-11T18:05:01.332 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-11T18:05:01.347 INFO:tasks.cephadm:Teardown complete 2024-05-11T18:05:01.347 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d/qa/tasks/cephadm.py", line 2272, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d/qa/tasks/cephadm.py", line 1901, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d/qa/tasks/cephadm.py", line 838, in ceph_bootstrap bootstrap_remote.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 1: 'echo \'# created by cephadm\n/var/log/ceph/{{ fsid }}/*.log {\n rotate 100\n size 10G\n compress\n sharedscripts\n postrotate\n killall -q -1 {{ targets|join(\'"\'"\' \'"\'"\') }} || pkill -1 -x \'"\'"\'{{ targets|join(\'"\'"\'|\'"\'"\') }}\'"\'"\' || true\n endscript\n missingok\n notifempty\n su root root\n}\' > /root/logrotate.conf' 2024-05-11T18:05:01.551 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1e55bfa4b6a84e838fa86b6699b945d8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d/qa/tasks/cephadm.py", line 2272, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d/qa/tasks/cephadm.py", line 1901, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_bfa64d6a7fc80d2d2d6f6aef3121ba27edfc8f2d/qa/tasks/cephadm.py", line 838, in ceph_bootstrap bootstrap_remote.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 1: 'echo \'# created by cephadm\n/var/log/ceph/{{ fsid }}/*.log {\n rotate 100\n size 10G\n compress\n sharedscripts\n postrotate\n killall -q -1 {{ targets|join(\'"\'"\' \'"\'"\') }} || pkill -1 -x \'"\'"\'{{ targets|join(\'"\'"\'|\'"\'"\') }}\'"\'"\' || true\n endscript\n missingok\n notifempty\n su root root\n}\' > /root/logrotate.conf' 2024-05-11T18:05:01.554 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-11T18:05:01.564 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-11T18:05:01.603 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-11T18:05:01.603 DEBUG:teuthology.orchestra.run.smithi044:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-11T18:05:01.621 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-11T18:05:01.628 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-05-11T18:05:01.632 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-11T18:05:01.632 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-05-11T18:05:01.632 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv01.front.sepia.ceph.com 3 6 37 40 +727us[ +727us] +/- 68ms 2024-05-11T18:05:01.632 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv02.front.sepia.ceph.com 3 6 37 40 -1106us[-1106us] +/- 80ms 2024-05-11T18:05:01.632 INFO:teuthology.orchestra.run.smithi044.stdout:^* hv03.front.sepia.ceph.com 3 6 37 40 +279us[ +254us] +/- 65ms 2024-05-11T18:05:01.633 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 7 0 - +0ns[ +0ns] +/- 0ns 2024-05-11T18:05:01.636 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-05-11T18:05:01.640 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-11T18:05:01.640 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-05-11T18:05:01.641 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv01.front.sepia.ceph.com 3 6 37 40 +770us[ +747us] +/- 68ms 2024-05-11T18:05:01.641 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv02.front.sepia.ceph.com 3 6 37 39 -1091us[-1091us] +/- 80ms 2024-05-11T18:05:01.641 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv03.front.sepia.ceph.com 3 6 37 39 +250us[ +227us] +/- 65ms 2024-05-11T18:05:01.641 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 7 0 - +0ns[ +0ns] +/- 0ns 2024-05-11T18:05:01.641 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-11T18:05:01.650 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-11T18:05:01.650 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-11T18:05:01.659 DEBUG:teuthology.orchestra.run.smithi044:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-11T18:05:01.739 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-11T18:05:01.794 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-11T18:05:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-11T18:05:01.824 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-11T18:05:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-11T18:05:01.851 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-11T18:05:01.859 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-11T18:05:01.870 INFO:teuthology.task.internal:Duration was 355.155943 seconds 2024-05-11T18:05:01.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-11T18:05:01.878 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-11T18:05:01.879 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-11T18:05:01.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-11T18:05:01.921 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-11T18:05:01.929 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-11T18:05:02.122 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-11T18:05:02.122 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi044.front.sepia.ceph.com 2024-05-11T18:05:02.122 DEBUG:teuthology.orchestra.run.smithi044:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-11T18:05:02.143 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2024-05-11T18:05:02.144 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-11T18:05:02.186 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-11T18:05:02.187 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-11T18:05:02.189 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-11T18:05:02.253 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-11T18:05:02.253 DEBUG:teuthology.orchestra.run.smithi044:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-11T18:05:02.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-11T18:05:02.381 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-11T18:05:02.390 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-11T18:05:02.390 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-11T18:05:02.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-11T18:05:02.441 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-11T18:05:02.450 DEBUG:teuthology.orchestra.run.smithi044:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-11T18:05:02.459 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-11T18:05:02.480 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = core 2024-05-11T18:05:02.504 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2024-05-11T18:05:02.521 DEBUG:teuthology.orchestra.run.smithi044:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-11T18:05:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-11T18:05:02.556 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-11T18:05:02.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-11T18:05:02.576 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-11T18:05:02.585 INFO:teuthology.task.internal:Transferring archived files... 2024-05-11T18:05:02.586 DEBUG:teuthology.misc:Transferring archived files from smithi044:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525/remote/smithi044 2024-05-11T18:05:02.586 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-11T18:05:02.633 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525/remote/smithi092 2024-05-11T18:05:02.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-11T18:05:02.666 INFO:teuthology.task.internal:Removing archive directory... 2024-05-11T18:05:02.666 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-11T18:05:02.670 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-11T18:05:02.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-11T18:05:02.727 INFO:teuthology.task.internal:Not uploading archives. 2024-05-11T18:05:02.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-11T18:05:02.736 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-11T18:05:02.737 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-11T18:05:02.739 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-11T18:05:02.752 INFO:teuthology.orchestra.run.smithi044.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 11 18:05 /home/ubuntu/cephtest 2024-05-11T18:05:02.770 INFO:teuthology.orchestra.run.smithi092.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 11 18:05 /home/ubuntu/cephtest 2024-05-11T18:05:02.771 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-11T18:05:02.782 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-11T18:05:02.800 INFO:teuthology.nuke:Checking targets against current locks 2024-05-11T18:05:02.827 DEBUG:teuthology.nuke:shortname: smithi044 2024-05-11T18:05:02.828 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-11T18:05:02.851 DEBUG:teuthology.nuke:shortname: smithi092 2024-05-11T18:05:02.851 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-11T18:05:02.855 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-11 17:50:38.097447', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEEys4usNeZqnyyizNgtpnTTuKdeUwlcF3HVdv426r+G4+HyhS7pGckEteU+20uRvl/HpEzhzNMqltPQpD+ulMo='} 2024-05-11T18:05:02.878 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-05-11_17:17:13-orch:cephadm-wip-adk-testing-2024-05-10-1625-distro-default-smithi/7702525', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-11 17:50:38.098640', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBrq2hnqrZLO9TGqrGty7x++2ljkeMfclJZgbr01L8+wWM2wo4MLcdq4t+KbbZvoDRMLfs1ZdSjR+sff2ff1T8c='} 2024-05-11T18:05:02.910 INFO:teuthology.orchestra.console.smithi044:Power off 2024-05-11T18:05:02.910 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-11T18:05:02.930 INFO:teuthology.orchestra.console.smithi092:Power off 2024-05-11T18:05:02.931 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-11T18:05:02.940 DEBUG:teuthology.orchestra.console.smithi044:power off output: Chassis Power Control: Down/Off 2024-05-11T18:05:02.940 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-11T18:05:02.953 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off 2024-05-11T18:05:02.954 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-11T18:05:02.965 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-05-11T18:05:02.977 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-05-11T18:05:06.966 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-11T18:05:06.978 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-11T18:05:07.076 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-05-11T18:05:07.087 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-05-11T18:05:11.082 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-11T18:05:11.090 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-11T18:05:11.192 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-05-11T18:05:11.199 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-05-11T18:05:15.194 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-11T18:05:15.202 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-11T18:05:15.304 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off 2024-05-11T18:05:15.312 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off 2024-05-11T18:05:15.405 INFO:teuthology.orchestra.console.smithi044:Power off completed 2024-05-11T18:05:15.413 INFO:teuthology.orchestra.console.smithi092:Power off completed 2024-05-11T18:05:15.518 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 355.1559433937073 failure_reason: 'Command failed on smithi044 with status 1: ''echo \''# created by cephadm\n/var/log/ceph/{{ fsid }}/*.log {\n rotate 100\n size 10G\n compress\n sharedscripts\n postrotate\n killall -q -1 {{ targets|join(\''"\''"\'' \''"\''"\'') }} || pkill -1 -x \''"\''"\''{{ targets|join(\''"\''"\''|\''"\''"\'') }}\''"\''"\'' || true\n endscript\n missingok\n notifempty\n su root root\n}\'' > /root/logrotate.conf''' owner: scheduled_adking@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1e55bfa4b6a84e838fa86b6699b945d8 status: fail success: false 2024-05-11T18:05:15.519 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-11T18:05:15.599 INFO:teuthology.run:FAIL