2024-04-22T21:24:09.249 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T21:24:09.249 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --owner scheduled_teuthology@teuthology --verbose --name teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751 -- /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751/orig.config.yaml 2024-04-22T21:24:09.284 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T21:24:09.371 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7668751' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 207 sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f sleep_before_teardown: 0 subset: 24/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9 targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJkiwdunrL3ypsH8NjMiQ8PcUtXSDIJO735kDxInWme+Xmszh8o6zvZNQXDtCDYozAkBA/wKhluGoJkN5ljbcJQ= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH3I00IjvBBGG1AjUkXfAHZHTfGaFHjiN4OLYSrQkywb/izUmEIwfuYwh5cAo/xJadCM4yDX+ZrkjyP1OFLX4l8= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-04-22T21:24:09.371 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa; will attempt to use it 2024-04-22T21:24:09.372 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks 2024-04-22T21:24:09.372 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T21:24:09.374 INFO:teuthology.task.internal:Checking packages... 2024-04-22T21:24:09.399 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '430e09df97c8fc7dc2b2ae424f68ed11366c540f' 2024-04-22T21:24:09.399 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T21:24:09.399 INFO:teuthology.packaging:ref: None 2024-04-22T21:24:09.400 INFO:teuthology.packaging:tag: None 2024-04-22T21:24:09.400 INFO:teuthology.packaging:branch: squid 2024-04-22T21:24:09.400 INFO:teuthology.packaging:sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:24:09.400 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-22T21:24:09.607 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2414.g055915ee 2024-04-22T21:24:09.608 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T21:24:09.616 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T21:24:09.616 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T21:24:09.623 INFO:teuthology.task.internal:Saving configuration 2024-04-22T21:24:09.636 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T21:24:09.642 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T21:24:09.668 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:19:09.111611', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJkiwdunrL3ypsH8NjMiQ8PcUtXSDIJO735kDxInWme+Xmszh8o6zvZNQXDtCDYozAkBA/wKhluGoJkN5ljbcJQ='} 2024-04-22T21:24:09.692 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:19:09.113248', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH3I00IjvBBGG1AjUkXfAHZHTfGaFHjiN4OLYSrQkywb/izUmEIwfuYwh5cAo/xJadCM4yDX+ZrkjyP1OFLX4l8='} 2024-04-22T21:24:09.692 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T21:24:09.719 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-22T21:24:09.719 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-22T21:24:09.720 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T21:24:09.791 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb512e83f70>, signals=[15]) 2024-04-22T21:24:09.792 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T21:24:09.798 INFO:teuthology.task.internal:Opening connections... 2024-04-22T21:24:09.798 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-04-22T21:24:09.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T21:24:09.878 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T21:24:09.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T21:24:09.952 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T21:24:09.959 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-04-22T21:24:09.977 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-04-22T21:24:09.977 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-04-22T21:24:10.034 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-04-22T21:24:10.034 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2024-04-22T21:24:10.034 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-04-22T21:24:10.034 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-04-22T21:24:10.034 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2024-04-22T21:24:10.035 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2024-04-22T21:24:10.035 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T21:24:10.035 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-04-22T21:24:10.035 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2024-04-22T21:24:10.035 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T21:24:10.035 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-04-22T21:24:10.035 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T21:24:10.035 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T21:24:10.035 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T21:24:10.036 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-04-22T21:24:10.082 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-04-22T21:24:10.100 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-04-22T21:24:10.100 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-04-22T21:24:10.157 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2024-04-22T21:24:10.157 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="9" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="9" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el9" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:LOGO="fedora-logo-icon" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T21:24:10.158 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T21:24:10.159 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-04-22T21:24:10.181 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T21:24:10.189 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T21:24:10.195 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T21:24:10.195 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-04-22T21:24:10.198 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-04-22T21:24:10.215 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T21:24:10.221 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T21:24:10.221 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-04-22T21:24:10.255 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-04-22T21:24:10.292 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T21:24:10.348 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T21:24:10.393 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T21:24:10.393 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T21:24:10.393 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T21:24:10.394 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T21:24:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:24:10.410 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-04-22T21:24:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:24:10.411 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-04-22T21:24:10.467 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-437.el9.x86_64 2024-04-22T21:24:10.467 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.14.0-437.el9.x86_64 2024-04-22T21:24:10.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2024-04-22T21:24:10.469 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-437.el9.x86_64 2024-04-22T21:24:10.469 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.14.0-437.el9.x86_64 2024-04-22T21:24:10.469 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2024-04-22T21:24:13.092 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 8.1 MB/s | 8.0 MB 00:00 2024-04-22T21:24:14.334 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-04-22T21:24:14.951 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-22T21:24:16.202 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-22T21:24:19.643 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 5.8 MB/s | 6.2 MB 00:01 2024-04-22T21:24:21.644 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-04-22T21:24:23.737 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-22T21:24:26.844 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 891 kB/s | 6.2 MB 00:07 2024-04-22T21:24:28.461 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-04-22T21:24:29.044 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 5.1 kB/s | 1.7 kB 00:00 2024-04-22T21:24:30.901 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T21:24:30.901 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T21:24:30.993 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T21:24:30.994 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-22T21:24:30.994 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T21:24:31.112 DEBUG:teuthology.orchestra.run.smithi172:> echo no | sudo yum reinstall kernel || true 2024-04-22T21:24:31.537 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 21 MB 00:02 2024-04-22T21:24:31.653 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:02 ago on Mon 22 Apr 2024 09:24:29 PM UTC. 2024-04-22T21:24:31.782 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T21:24:31.790 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T21:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-04-22T21:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T21:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout:Reinstalling: 2024-04-22T21:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T21:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T21:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 255 k 2024-04-22T21:24:31.792 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 0 2024-04-22T21:24:31.792 INFO:teuthology.orchestra.run.smithi172.stderr:Operation aborted. 2024-04-22T21:24:31.855 INFO:teuthology.orchestra.run.smithi172.stdout:Is this ok [y/N]: 2024-04-22T21:24:31.856 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum reinstall -y kernel || true 2024-04-22T21:24:32.349 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 09:24:29 PM UTC. 2024-04-22T21:24:32.465 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T21:24:32.465 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T21:24:32.466 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-04-22T21:24:32.466 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T21:24:32.466 INFO:teuthology.orchestra.run.smithi172.stdout:Reinstalling: 2024-04-22T21:24:32.466 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:24:32.466 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:24:32.466 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T21:24:32.466 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T21:24:32.466 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:24:32.466 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 255 k 2024-04-22T21:24:32.466 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 0 2024-04-22T21:24:32.466 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T21:24:32.697 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-437.el9.x86_64.rpm 2.8 MB/s | 255 kB 00:00 2024-04-22T21:24:32.697 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T21:24:32.697 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.1 MB/s | 255 kB 00:00 2024-04-22T21:24:32.715 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T21:24:32.909 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T21:24:32.909 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T21:24:32.914 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T21:24:32.914 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T21:24:33.060 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T21:24:33.131 INFO:teuthology.orchestra.run.smithi172.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:24:33.131 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:24:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:24:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:24:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:24:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:24:34.528 INFO:teuthology.orchestra.run.smithi172.stdout:Reinstalled: 2024-04-22T21:24:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T21:24:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:24:34.528 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T21:24:34.693 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T21:24:34.740 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T21:24:34.740 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T21:24:34.740 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T21:24:34.740 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-22T21:24:34.740 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T21:24:34.770 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-04-22T21:24:34.793 DEBUG:teuthology.parallel:result is None 2024-04-22T21:24:36.587 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-22T21:24:38.410 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T21:24:38.410 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T21:24:38.480 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-22T21:24:38.481 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-04-22T21:24:38.481 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-22T21:24:38.580 DEBUG:teuthology.orchestra.run.smithi083:> echo no | sudo yum reinstall kernel || true 2024-04-22T21:24:39.080 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 09:24:36 PM UTC. 2024-04-22T21:24:39.228 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-22T21:24:39.233 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-22T21:24:39.233 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-04-22T21:24:39.233 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-22T21:24:39.233 INFO:teuthology.orchestra.run.smithi083.stdout:Reinstalling: 2024-04-22T21:24:39.233 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:24:39.233 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:24:39.233 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-22T21:24:39.233 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-22T21:24:39.233 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:24:39.233 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 255 k 2024-04-22T21:24:39.233 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 0 2024-04-22T21:24:39.234 INFO:teuthology.orchestra.run.smithi083.stderr:Operation aborted. 2024-04-22T21:24:39.287 INFO:teuthology.orchestra.run.smithi083.stdout:Is this ok [y/N]: 2024-04-22T21:24:39.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum reinstall -y kernel || true 2024-04-22T21:24:39.779 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 09:24:36 PM UTC. 2024-04-22T21:24:39.895 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-04-22T21:24:39.896 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-22T21:24:39.896 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-04-22T21:24:39.896 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-22T21:24:39.896 INFO:teuthology.orchestra.run.smithi083.stdout:Reinstalling: 2024-04-22T21:24:39.896 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:24:39.896 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:24:39.896 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-04-22T21:24:39.896 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-04-22T21:24:39.896 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:24:39.896 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 255 k 2024-04-22T21:24:39.896 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 0 2024-04-22T21:24:39.897 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-04-22T21:24:40.381 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.2 MB/s | 255 kB 00:00 2024-04-22T21:24:40.381 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-04-22T21:24:40.381 INFO:teuthology.orchestra.run.smithi083.stdout:Total 526 kB/s | 255 kB 00:00 2024-04-22T21:24:40.397 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-04-22T21:24:40.581 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-04-22T21:24:40.581 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-04-22T21:24:40.586 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-04-22T21:24:40.586 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-04-22T21:24:40.716 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-04-22T21:24:40.799 INFO:teuthology.orchestra.run.smithi083.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:24:40.799 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:24:41.495 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:24:41.496 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:24:42.219 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:24:42.219 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:24:42.220 INFO:teuthology.orchestra.run.smithi083.stdout:Reinstalled: 2024-04-22T21:24:42.220 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T21:24:42.220 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:24:42.220 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-04-22T21:24:42.386 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T21:24:42.433 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T21:24:42.433 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T21:24:42.434 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T21:24:42.434 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-22T21:24:42.434 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T21:24:42.470 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-04-22T21:24:42.495 DEBUG:teuthology.parallel:result is None 2024-04-22T21:24:42.496 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T21:24:42.504 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T21:24:42.504 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T21:24:42.538 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T21:24:42.556 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T21:24:42.563 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T21:24:42.570 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T21:24:42.571 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T21:24:42.593 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T21:24:42.693 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T21:24:42.700 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T21:24:42.700 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T21:24:42.703 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T21:24:42.732 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:24:42.735 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:24:42.743 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:24:42.744 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:24:42.746 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T21:24:42.768 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T21:24:42.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T21:24:42.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T21:24:42.816 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T21:24:42.836 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T21:24:42.837 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T21:24:42.852 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T21:24:42.873 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:24:42.947 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:24:42.996 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:24:43.069 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:24:43.116 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-22T21:24:43.116 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T21:24:43.179 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:24:43.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:24:43.269 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:24:43.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:24:43.394 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T21:24:43.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T21:24:43.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-04-22T21:24:43.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-04-22T21:24:43.490 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T21:24:43.542 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T21:24:43.936 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T21:24:43.944 INFO:teuthology.task.internal:Starting timer... 2024-04-22T21:24:43.944 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T21:24:43.955 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T21:24:43.963 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-22T21:24:43.964 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-04-22T21:24:44.013 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-04-22T21:24:44.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2024-04-22T21:24:44.084 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2024-04-22T21:24:44.086 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T21:24:44.086 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-04-22T21:24:44.105 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-04-22T21:24:44.105 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2024-04-22T21:24:44.124 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2024-04-22T21:24:44.124 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi083.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2024-04-22T21:24:44.124 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T21:24:44.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:24:44.170 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T21:24:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:24:44.195 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T21:24:44.195 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-04-22T21:24:44.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2024-04-22T21:24:44.261 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T21:24:44.278 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T21:24:44.298 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T21:24:44.299 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T21:28:14.179 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-04-22T21:28:14.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-04-22T21:28:14.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T21:28:14.260 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-04-22T21:28:14.337 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-04-22T21:28:14.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-04-22T21:28:14.338 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T21:28:14.414 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-04-22T21:28:14.489 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-04-22T21:28:14.489 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T21:28:14.500 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T21:28:14.500 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T21:28:14.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T21:28:14.504 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T21:28:14.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T21:28:14.530 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T21:28:14.543 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T21:28:14.556 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T21:28:14.565 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-04-22T21:28:14.570 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T21:28:14.575 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-04-22T21:28:14.589 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T21:28:14.592 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2024-04-22T21:28:14.600 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T21:28:14.603 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2024-04-22T21:28:14.616 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T21:28:14.628 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T21:28:14.652 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-04-22T21:28:14.656 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T21:28:14.656 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-04-22T21:28:14.656 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:28:14.656 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:28:14.656 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:28:14.656 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:28:14.684 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-04-22T21:28:14.687 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T21:28:14.687 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-04-22T21:28:14.687 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:28:14.687 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:28:14.687 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:28:14.687 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:28:14.688 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T21:28:14.793 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '430e09df97c8fc7dc2b2ae424f68ed11366c540f'} 2024-04-22T21:28:14.793 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-22T21:28:14.794 INFO:tasks.cephadm:Cluster fsid is 3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:28:14.794 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T21:28:14.794 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-22T21:28:14.794 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi083': '172.21.15.83', 'mon.smithi172': '172.21.15.172'} 2024-04-22T21:28:14.794 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T21:28:14.795 DEBUG:teuthology.orchestra.run.smithi083:> sudo hostname $(hostname -s) 2024-04-22T21:28:14.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo hostname $(hostname -s) 2024-04-22T21:28:14.846 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-22T21:28:14.846 DEBUG:teuthology.orchestra.run.smithi083:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T21:28:15.020 INFO:teuthology.orchestra.run.smithi083.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 22 21:28 /home/ubuntu/cephtest/cephadm 2024-04-22T21:28:15.021 DEBUG:teuthology.orchestra.run.smithi172:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T21:28:15.247 INFO:teuthology.orchestra.run.smithi172.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 22 21:28 /home/ubuntu/cephtest/cephadm 2024-04-22T21:28:15.248 DEBUG:teuthology.orchestra.run.smithi083:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T21:28:15.264 DEBUG:teuthology.orchestra.run.smithi172:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T21:28:15.288 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-22T21:28:15.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-22T21:28:15.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-22T21:28:16.890 INFO:teuthology.orchestra.run.smithi172.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T21:28:16.903 INFO:teuthology.orchestra.run.smithi083.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T21:28:37.641 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-04-22T21:28:37.641 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-22T21:28:37.642 INFO:teuthology.orchestra.run.smithi172.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-22T21:28:37.642 INFO:teuthology.orchestra.run.smithi172.stdout: "repo_digests": [ 2024-04-22T21:28:37.642 INFO:teuthology.orchestra.run.smithi172.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-22T21:28:37.642 INFO:teuthology.orchestra.run.smithi172.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-22T21:28:37.642 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-04-22T21:28:37.642 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-04-22T21:28:39.585 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-04-22T21:28:39.585 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-22T21:28:39.585 INFO:teuthology.orchestra.run.smithi083.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-22T21:28:39.585 INFO:teuthology.orchestra.run.smithi083.stdout: "repo_digests": [ 2024-04-22T21:28:39.586 INFO:teuthology.orchestra.run.smithi083.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-22T21:28:39.586 INFO:teuthology.orchestra.run.smithi083.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-22T21:28:39.586 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-04-22T21:28:39.586 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-04-22T21:28:39.604 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph 2024-04-22T21:28:39.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph 2024-04-22T21:28:39.698 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /etc/ceph 2024-04-22T21:28:39.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /etc/ceph 2024-04-22T21:28:39.766 INFO:tasks.cephadm:Writing seed config... 2024-04-22T21:28:39.767 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T21:28:39.768 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T21:28:39.768 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T21:28:39.768 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T21:28:39.768 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T21:28:39.768 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T21:28:39.768 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T21:28:39.768 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-22T21:28:39.769 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-22T21:28:39.769 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T21:28:39.783 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3a5a1c14-00ef-11ef-bc93-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-22T21:28:39.784 DEBUG:teuthology.orchestra.run.smithi083:mon.smithi083> sudo journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mon.smithi083.service 2024-04-22T21:28:39.826 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T21:28:39.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.83 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:-------------------------------------------------------------------------------- 2024-04-22T21:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '3a5a1c14-00ef-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.83', '--skip-admin-label'] 2024-04-22T21:28:40.041 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: 4.9.4-dev 2024-04-22T21:28:40.044 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying podman|docker is present... 2024-04-22T21:28:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: 4.9.4-dev 2024-04-22T21:28:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying lvm2 is present... 2024-04-22T21:28:40.068 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying time synchronization is in place... 2024-04-22T21:28:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T21:28:40.080 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-04-22T21:28:40.087 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: enabled 2024-04-22T21:28:40.092 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: active 2024-04-22T21:28:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:Unit chronyd.service is enabled and running 2024-04-22T21:28:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:Repeating the final host check... 2024-04-22T21:28:40.114 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: 4.9.4-dev 2024-04-22T21:28:40.116 INFO:teuthology.orchestra.run.smithi083.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-22T21:28:40.116 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl is present 2024-04-22T21:28:40.116 INFO:teuthology.orchestra.run.smithi083.stderr:lvcreate is present 2024-04-22T21:28:40.122 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T21:28:40.128 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-04-22T21:28:40.134 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: enabled 2024-04-22T21:28:40.140 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: active 2024-04-22T21:28:40.141 INFO:teuthology.orchestra.run.smithi083.stderr:Unit chronyd.service is enabled and running 2024-04-22T21:28:40.141 INFO:teuthology.orchestra.run.smithi083.stderr:Host looks OK 2024-04-22T21:28:40.141 INFO:teuthology.orchestra.run.smithi083.stderr:Cluster fsid: 3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:28:40.141 INFO:teuthology.orchestra.run.smithi083.stderr:Acquiring lock 140280993948960 on /run/cephadm/3a5a1c14-00ef-11ef-bc93-c7b262605968.lock 2024-04-22T21:28:40.141 INFO:teuthology.orchestra.run.smithi083.stderr:Lock 140280993948960 acquired on /run/cephadm/3a5a1c14-00ef-11ef-bc93-c7b262605968.lock 2024-04-22T21:28:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying IP 172.21.15.83 port 3300 ... 2024-04-22T21:28:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying IP 172.21.15.83 port 6789 ... 2024-04-22T21:28:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:Base mon IP is 172.21.15.83, final addrv is [v2:172.21.15.83:3300,v1:172.21.15.83:6789] 2024-04-22T21:28:40.144 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.83 metric 100 2024-04-22T21:28:40.144 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.83 metric 100 2024-04-22T21:28:40.147 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T21:28:40.147 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-22T21:28:40.147 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-22T21:28:40.149 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T21:28:40.149 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-22T21:28:40.149 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-22T21:28:40.149 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-22T21:28:40.150 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: inet6 fe80::caae:46bb:752a:fdb5/64 scope link noprefixroute 2024-04-22T21:28:40.150 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-22T21:28:40.150 INFO:teuthology.orchestra.run.smithi083.stderr:Mon IP `172.21.15.83` is in CIDR network `172.21.0.0/20` 2024-04-22T21:28:40.150 INFO:teuthology.orchestra.run.smithi083.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T21:28:40.151 INFO:teuthology.orchestra.run.smithi083.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T21:28:40.252 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-22T21:28:40.616 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Getting image source signatures 2024-04-22T21:28:40.616 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-22T21:28:40.616 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-22T21:28:40.616 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-22T21:28:40.616 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-22T21:28:40.616 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-22T21:28:40.616 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-22T21:28:40.788 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Writing manifest to image destination 2024-04-22T21:28:40.933 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-22T21:28:41.351 INFO:teuthology.orchestra.run.smithi083.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-22T21:28:41.875 INFO:teuthology.orchestra.run.smithi083.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-22T21:28:41.875 INFO:teuthology.orchestra.run.smithi083.stderr:Extracting ceph user uid/gid from container image... 2024-04-22T21:28:42.265 INFO:teuthology.orchestra.run.smithi083.stderr:stat: 167 167 2024-04-22T21:28:42.904 INFO:teuthology.orchestra.run.smithi083.stderr:Creating initial keys... 2024-04-22T21:28:43.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-authtool: AQCL1iZmTm2JDxAAagYz/JUuVwFpkmR7hu0flw== 2024-04-22T21:28:44.176 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-authtool: AQCM1iZmL/52BhAAB/Ooln4npzdU0/SiHKu/rg== 2024-04-22T21:28:45.057 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-authtool: AQCM1iZm3HBoOhAAWhrT6/CXbfBgxASCW8c0Wg== 2024-04-22T21:28:45.506 INFO:teuthology.orchestra.run.smithi083.stderr:Creating initial monmap... 2024-04-22T21:28:45.880 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T21:28:45.880 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-22T21:28:45.880 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:28:45.880 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T21:28:46.420 INFO:teuthology.orchestra.run.smithi083.stderr:monmaptool for smithi083 [v2:172.21.15.83:3300,v1:172.21.15.83:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T21:28:46.420 INFO:teuthology.orchestra.run.smithi083.stderr:setting min_mon_release = octopus 2024-04-22T21:28:46.420 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: set fsid to 3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:28:46.420 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T21:28:46.420 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-22T21:28:46.420 INFO:teuthology.orchestra.run.smithi083.stderr:Creating mon... 2024-04-22T21:28:47.368 INFO:teuthology.orchestra.run.smithi083.stderr:create mon.smithi083 on 2024-04-22T21:28:47.584 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T21:28:47.873 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968.target. 2024-04-22T21:28:47.874 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968.target. 2024-04-22T21:28:48.165 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to reset failed state of unit ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mon.smithi083.service: Unit ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mon.smithi083.service not loaded. 2024-04-22T21:28:48.172 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968.target.wants/ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mon.smithi083.service → /etc/systemd/system/ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@.service. 2024-04-22T21:28:48.820 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:48 smithi083 ceph-mon[21445]: mon.smithi083 is new leader, mons smithi083 in quorum (ranks 0) 2024-04-22T21:28:48.832 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-04-22T21:28:48.839 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-04-22T21:28:48.840 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-04-22T21:28:48.840 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to enable service . firewalld.service is not available 2024-04-22T21:28:48.841 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mon to start... 2024-04-22T21:28:48.841 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mon... 2024-04-22T21:28:49.131 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:48 smithi083 bash[21425]: 1f22531db55d434b622c46a60568bfdc2cbdf572b9640e2053cf263cc57aae45 2024-04-22T21:28:49.131 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:48 smithi083 systemd[1]: Started Ceph mon.smithi083 for 3a5a1c14-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:28:49.303 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: cluster: 2024-04-22T21:28:49.303 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: id: 3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:28:49.303 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-22T21:28:49.303 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-04-22T21:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: services: 2024-04-22T21:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi083 (age 0.515259s) 2024-04-22T21:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-22T21:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-22T21:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-04-22T21:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: data: 2024-04-22T21:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-22T21:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-22T21:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-22T21:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: pgs: 2024-04-22T21:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-04-22T21:28:49.837 INFO:teuthology.orchestra.run.smithi083.stderr:mon is available 2024-04-22T21:28:49.837 INFO:teuthology.orchestra.run.smithi083.stderr:Assimilating anything we can from ceph.conf... 2024-04-22T21:28:49.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:49 smithi083 ceph-mon[21445]: mon.smithi083 is new leader, mons smithi083 in quorum (ranks 0) 2024-04-22T21:28:49.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:49 smithi083 ceph-mon[21445]: monmap e1: 1 mons at {smithi083=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0]} 2024-04-22T21:28:49.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:49 smithi083 ceph-mon[21445]: fsmap 2024-04-22T21:28:49.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:49 smithi083 ceph-mon[21445]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T21:28:49.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:49 smithi083 ceph-mon[21445]: mgrmap e1: no daemons active 2024-04-22T21:28:49.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:49 smithi083 ceph-mon[21445]: from='client.? 172.21.15.83:0/4080939206' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T21:28:50.360 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-04-22T21:28:50.360 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [global] 2024-04-22T21:28:50.360 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: fsid = 3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:28:50.360 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.83:3300,v1:172.21.15.83:6789] 2024-04-22T21:28:50.360 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-22T21:28:50.360 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-22T21:28:50.360 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-22T21:28:50.360 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-22T21:28:50.360 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-04-22T21:28:50.361 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [mgr] 2024-04-22T21:28:50.361 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-22T21:28:50.361 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-04-22T21:28:50.361 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [osd] 2024-04-22T21:28:50.361 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-22T21:28:50.361 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-22T21:28:50.902 INFO:teuthology.orchestra.run.smithi083.stderr:Generating new minimal ceph.conf... 2024-04-22T21:28:51.017 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:50 smithi083 ceph-mon[21445]: from='client.? 172.21.15.83:0/2084378937' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T21:28:51.018 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:50 smithi083 ceph-mon[21445]: from='client.? 172.21.15.83:0/2084378937' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T21:28:51.889 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:51 smithi083 ceph-mon[21445]: from='client.? 172.21.15.83:0/3656939276' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:28:51.892 INFO:teuthology.orchestra.run.smithi083.stderr:Restarting the monitor... 2024-04-22T21:28:52.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:51 smithi083 systemd[1]: Stopping Ceph mon.smithi083 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:28:52.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:51 smithi083 bash[21662]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon.smithi083" found: no such container 2024-04-22T21:28:52.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083[21441]: 2024-04-22T21:28:52.066+0000 7f5a9f42a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi083 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:28:52.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083[21441]: 2024-04-22T21:28:52.066+0000 7f5a9f42a700 -1 mon.smithi083@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T21:28:52.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 podman[21675]: 2024-04-22 21:28:52.087438845 +0000 UTC m=+0.125516505 container died 1f22531db55d434b622c46a60568bfdc2cbdf572b9640e2053cf263cc57aae45 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083, com.redhat.component=centos-stream-container, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , ceph=True, name=centos-stream, distribution-scope=public, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T21:28:52.643 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 podman[21675]: 2024-04-22 21:28:52.458473675 +0000 UTC m=+0.496551337 container cleanup 1f22531db55d434b622c46a60568bfdc2cbdf572b9640e2053cf263cc57aae45 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, release=754, io.buildah.version=1.19.8, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, vcs-type=git, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, name=centos-stream) 2024-04-22T21:28:52.643 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 bash[21675]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083 2024-04-22T21:28:52.643 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 bash[21701]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon.smithi083" found: no such container 2024-04-22T21:28:52.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 podman[21689]: 2024-04-22 21:28:52.659587891 +0000 UTC m=+0.567542385 container remove 1f22531db55d434b622c46a60568bfdc2cbdf572b9640e2053cf263cc57aae45 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083, ceph=True, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=) 2024-04-22T21:28:52.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 systemd[1]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mon.smithi083.service: Deactivated successfully. 2024-04-22T21:28:52.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 systemd[1]: Stopped Ceph mon.smithi083 for 3a5a1c14-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:28:52.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 systemd[1]: Starting Ceph mon.smithi083 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:28:52.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 podman[21769]: 2024-04-22 21:28:52.876933786 +0000 UTC m=+0.107888401 container create 25f27fe2e2e709a1032e4f2d5acc8c0372c682ba4d413cbae05a8ea22f7d4ca2 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, ceph=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, release=754, com.redhat.component=centos-stream-container, name=centos-stream, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git) 2024-04-22T21:28:52.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:52 smithi083 podman[21769]: 2024-04-22 21:28:52.78853127 +0000 UTC m=+0.019485906 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-22T21:28:53.133 INFO:teuthology.orchestra.run.smithi083.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-22T21:28:53.247 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 podman[21769]: 2024-04-22 21:28:53.079354981 +0000 UTC m=+0.310309596 container init 25f27fe2e2e709a1032e4f2d5acc8c0372c682ba4d413cbae05a8ea22f7d4ca2 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083, name=centos-stream, ceph=True, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, distribution-scope=public, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.expose-services=, architecture=x86_64, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T21:28:53.248 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 podman[21769]: 2024-04-22 21:28:53.082220571 +0000 UTC m=+0.313175192 container start 25f27fe2e2e709a1032e4f2d5acc8c0372c682ba4d413cbae05a8ea22f7d4ca2 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083, GIT_BRANCH=HEAD, distribution-scope=public, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, ceph=True, vcs-type=git, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., name=centos-stream, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.buildah.version=1.19.8, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-22T21:28:53.248 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T21:28:53.248 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-22T21:28:53.248 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: pidfile_write: ignore empty --pid-file 2024-04-22T21:28:53.248 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: load: jerasure load: lrc 2024-04-22T21:28:53.248 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: RocksDB version: 6.15.5 2024-04-22T21:28:53.248 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-22T21:28:53.248 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Compile date Apr 18 2022 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: DB SUMMARY 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: DB Session ID: C3G4ITBOY01DMRR30DGM 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: CURRENT file: CURRENT 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi083/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi083/store.db: 000010.log size: 82713 ; 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.error_if_exists: 0 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.create_if_missing: 0 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.paranoid_checks: 1 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.env: 0x563b6b0c6860 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.fs: Posix File System 2024-04-22T21:28:53.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.info_log: 0x563b6c5f3fa0 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.statistics: (nil) 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.use_fsync: 0 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_log_file_size: 0 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.allow_fallocate: 1 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.use_direct_reads: 0 2024-04-22T21:28:53.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T21:28:53.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T21:28:53.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.db_log_dir: 2024-04-22T21:28:53.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi083/store.db 2024-04-22T21:28:53.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T21:28:53.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T21:28:53.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T21:28:53.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T21:28:53.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T21:28:53.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T21:28:53.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T21:28:53.252 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T21:28:53.252 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.write_buffer_manager: 0x563b6c6e4330 2024-04-22T21:28:53.252 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.unordered_write: 0 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.row_cache: None 2024-04-22T21:28:53.253 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.wal_filter: None 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.preserve_deletes: 0 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.two_write_queues: 0 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.atomic_flush: 0 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.log_readahead_size: 0 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T21:28:53.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_background_jobs: 2 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_background_compactions: -1 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_subcompactions: 1 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_open_files: -1 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T21:28:53.255 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_background_flushes: -1 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Compression algorithms supported: 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: kZSTD supported: 0 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: kXpressCompression supported: 0 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: kLZ4Compression supported: 1 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: kBZip2Compression supported: 0 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: kZlibCompression supported: 1 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: kSnappyCompression supported: 1 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T21:28:53.256 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi083/store.db/MANIFEST-000009 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.merge_operator: 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_filter: None 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563b6c5bfd00) 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: cache_index_and_filter_blocks: 1 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: pin_top_level_index_and_filter: 1 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: index_type: 0 2024-04-22T21:28:53.257 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: data_block_index_type: 0 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: index_shortening: 1 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: hash_index_allow_collision: 1 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: checksum: 1 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: no_block_cache: 0 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_cache: 0x563b6c62a170 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_cache_name: BinnedLRUCache 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_cache_options: 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: capacity : 536870912 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: num_shard_bits : 4 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: strict_capacity_limit : 0 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: high_pri_pool_ratio: 0.000 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_cache_compressed: (nil) 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: persistent_cache: (nil) 2024-04-22T21:28:53.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_size: 4096 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_size_deviation: 10 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_restart_interval: 16 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: index_block_restart_interval: 1 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: metadata_block_size: 4096 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: partition_filters: 0 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: use_delta_encoding: 1 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: whole_key_filtering: 1 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: verify_compression: 0 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: read_amp_bytes_per_bit: 0 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: format_version: 4 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: enable_index_compression: 1 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_align: 0 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T21:28:53.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compression: NoCompression 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.num_levels: 7 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T21:28:53.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T21:28:53.261 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T21:28:53.261 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T21:28:53.261 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T21:28:53.261 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:28:53.261 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T21:28:53.261 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T21:28:53.261 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T21:28:53.261 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T21:28:53.261 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T21:28:53.261 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T21:28:53.261 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.arena_block_size: 4194304 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T21:28:53.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.table_properties_collectors: 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.inplace_update_support: 0 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T21:28:53.263 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.bloom_locality: 0 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.max_successive_merges: 0 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.ttl: 2592000 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.enable_blob_files: false 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.min_blob_size: 0 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T21:28:53.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi083/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821333107392, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821333108085, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79395, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77659, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71968, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713821333, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "cdc81fa6-5648-4049-9c83-171a6655f85f", "db_session_id": "C3G4ITBOY01DMRR30DGM"}} 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821333108413, "job": 1, "event": "recovery_finished"} 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi083/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x563b6c610a80 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: DB pointer 0x563b6c620000 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-22T21:28:53.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ** DB Stats ** 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ** Compaction Stats [default] ** 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: L0 2/0 79.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.8 0.00 0.00 1 0.000 0 0 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Sum 2/0 79.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.8 0.00 0.00 1 0.000 0 0 2024-04-22T21:28:53.266 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.8 0.00 0.00 1 0.000 0 0 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ** Compaction Stats [default] ** 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 162.8 0.00 0.00 1 0.000 0 0 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Cumulative compaction: 0.00 GB write, 26.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Interval compaction: 0.00 GB write, 26.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:28:53.267 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ** Compaction Stats [default] ** 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: L0 2/0 79.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.8 0.00 0.00 1 0.000 0 0 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Sum 2/0 79.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.8 0.00 0.00 1 0.000 0 0 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ** Compaction Stats [default] ** 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 162.8 0.00 0.00 1 0.000 0 0 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:28:53.268 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Cumulative compaction: 0.00 GB write, 25.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: starting mon.smithi083 rank 0 at public addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] at bind addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi083 fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: mon.smithi083@-1(???) e1 preinit fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: mon.smithi083@-1(???).mds e1 new map 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: mon.smithi083@-1(???).mds e1 print_map 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: e1 2024-04-22T21:28:53.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T21:28:53.270 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T21:28:53.270 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: legacy client fscid: -1 2024-04-22T21:28:53.270 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-04-22T21:28:53.270 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: No filesystems configured 2024-04-22T21:28:53.270 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: mon.smithi083@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T21:28:53.270 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: mon.smithi083@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:28:53.270 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: mon.smithi083@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:28:53.270 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: mon.smithi083@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:28:53.270 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: mon.smithi083@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T21:28:53.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta expand map: {default=false} 2024-04-22T21:28:53.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta from 'false' to 'false' 2024-04-22T21:28:53.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta expanded map: {default=false} 2024-04-22T21:28:53.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta expand map: {default=info} 2024-04-22T21:28:53.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta from 'info' to 'info' 2024-04-22T21:28:53.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta expanded map: {default=info} 2024-04-22T21:28:53.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta expand map: {default=daemon} 2024-04-22T21:28:53.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-22T21:28:53.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta expanded map: {default=daemon} 2024-04-22T21:28:53.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta expand map: {default=debug} 2024-04-22T21:28:53.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta from 'debug' to 'debug' 2024-04-22T21:28:53.272 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: expand_channel_meta expanded map: {default=debug} 2024-04-22T21:28:53.272 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 bash[21769]: 25f27fe2e2e709a1032e4f2d5acc8c0372c682ba4d413cbae05a8ea22f7d4ca2 2024-04-22T21:28:53.272 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 systemd[1]: Started Ceph mon.smithi083 for 3a5a1c14-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:28:53.272 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: mon.smithi083 is new leader, mons smithi083 in quorum (ranks 0) 2024-04-22T21:28:53.272 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: monmap e1: 1 mons at {smithi083=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0]} 2024-04-22T21:28:53.272 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: fsmap 2024-04-22T21:28:53.272 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T21:28:53.272 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:53 smithi083 ceph-mon[21788]: mgrmap e1: no daemons active 2024-04-22T21:28:54.223 INFO:teuthology.orchestra.run.smithi083.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-22T21:28:54.224 INFO:teuthology.orchestra.run.smithi083.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:28:54.224 INFO:teuthology.orchestra.run.smithi083.stderr:Creating mgr... 2024-04-22T21:28:54.224 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying port 9283 ... 2024-04-22T21:28:54.388 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to reset failed state of unit ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mgr.smithi083.ochghz.service: Unit ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mgr.smithi083.ochghz.service not loaded. 2024-04-22T21:28:54.396 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968.target.wants/ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mgr.smithi083.ochghz.service → /etc/systemd/system/ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@.service. 2024-04-22T21:28:54.991 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:54 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/2846820671' entity='client.admin' 2024-04-22T21:28:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-04-22T21:28:55.114 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-04-22T21:28:55.115 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-04-22T21:28:55.115 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to enable service . firewalld.service is not available 2024-04-22T21:28:55.124 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-04-22T21:28:55.132 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-04-22T21:28:55.133 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-04-22T21:28:55.133 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-22T21:28:55.133 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr to start... 2024-04-22T21:28:55.133 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr... 2024-04-22T21:28:55.627 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsid": "3a5a1c14-00ef-11ef-bc93-c7b262605968", 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "health": { 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 0 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-04-22T21:28:55.628 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "smithi083" 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T21:28:55.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T21:28:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T21:28:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:28:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": false, 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "iostat", 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "nfs", 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "restful" 2024-04-22T21:28:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-04-22T21:28:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-04-22T21:28:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:28:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T21:28:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:28:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modified": "2024-04-22T21:28:48.804236+0000", 2024-04-22T21:28:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-04-22T21:28:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:28:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T21:28:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-04-22T21:28:55.800 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:28:55 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/833212458' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T21:29:00.361 INFO:teuthology.orchestra.run.smithi083.stderr:mgr not available, waiting (1/15)... 2024-04-22T21:29:01.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: Activating manager daemon smithi083.ochghz 2024-04-22T21:29:01.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: mgrmap e2: smithi083.ochghz(active, starting, since 0.00275286s) 2024-04-22T21:29:01.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: from='mgr.14100 172.21.15.83:0/2937829550' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:29:01.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: from='mgr.14100 172.21.15.83:0/2937829550' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:29:01.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: from='mgr.14100 172.21.15.83:0/2937829550' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:29:01.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: from='mgr.14100 172.21.15.83:0/2937829550' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-04-22T21:29:01.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: from='mgr.14100 172.21.15.83:0/2937829550' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mgr metadata", "who": "smithi083.ochghz", "id": "smithi083.ochghz"}]: dispatch 2024-04-22T21:29:01.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: Manager daemon smithi083.ochghz is now available 2024-04-22T21:29:01.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: from='mgr.14100 172.21.15.83:0/2937829550' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:29:01.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: from='mgr.14100 172.21.15.83:0/2937829550' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:29:01.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: from='mgr.14100 172.21.15.83:0/2937829550' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:01.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: from='mgr.14100 172.21.15.83:0/2937829550' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:01.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:01 smithi083 ceph-mon[21788]: from='mgr.14100 172.21.15.83:0/2937829550' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:03.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:02 smithi083 ceph-mon[21788]: mgrmap e3: smithi083.ochghz(active, since 1.00682s) 2024-04-22T21:29:03.705 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-04-22T21:29:03.705 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-04-22T21:29:03.705 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsid": "3a5a1c14-00ef-11ef-bc93-c7b262605968", 2024-04-22T21:29:03.705 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "health": { 2024-04-22T21:29:03.705 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T21:29:03.705 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T21:29:03.705 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T21:29:03.706 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:29:03.706 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T21:29:03.706 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T21:29:03.706 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 0 2024-04-22T21:29:03.706 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-04-22T21:29:03.706 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T21:29:03.706 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "smithi083" 2024-04-22T21:29:03.706 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T21:29:03.707 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T21:29:03.712 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T21:29:03.712 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T21:29:03.712 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:29:03.712 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T21:29:03.712 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T21:29:03.712 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T21:29:03.712 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T21:29:03.712 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T21:29:03.712 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T21:29:03.712 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": true, 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "iostat", 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "nfs", 2024-04-22T21:29:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "restful" 2024-04-22T21:29:03.714 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-04-22T21:29:03.714 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-04-22T21:29:03.714 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:29:03.714 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T21:29:03.714 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:29:03.714 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modified": "2024-04-22T21:28:48.804236+0000", 2024-04-22T21:29:03.714 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-04-22T21:29:03.714 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-04-22T21:29:03.714 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T21:29:03.714 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-04-22T21:29:04.239 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:03 smithi083 ceph-mon[21788]: mgrmap e4: smithi083.ochghz(active, since 2s) 2024-04-22T21:29:04.239 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:03 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/2195926638' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T21:29:04.241 INFO:teuthology.orchestra.run.smithi083.stderr:mgr is available 2024-04-22T21:29:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-04-22T21:29:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [global] 2024-04-22T21:29:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: fsid = 3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:29:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-22T21:29:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-22T21:29:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-22T21:29:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-22T21:29:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-04-22T21:29:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [mgr] 2024-04-22T21:29:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-22T21:29:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-04-22T21:29:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [osd] 2024-04-22T21:29:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-22T21:29:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-22T21:29:05.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:04 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/2317033761' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T21:29:05.289 INFO:teuthology.orchestra.run.smithi083.stderr:Enabling cephadm module... 2024-04-22T21:29:06.262 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:05 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1252385624' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T21:29:07.147 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-04-22T21:29:07.148 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-22T21:29:07.148 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": true, 2024-04-22T21:29:07.148 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "active_name": "smithi083.ochghz", 2024-04-22T21:29:07.148 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-22T21:29:07.148 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-04-22T21:29:07.403 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:06 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1252385624' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T21:29:07.404 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:06 smithi083 ceph-mon[21788]: mgrmap e5: smithi083.ochghz(active, since 5s) 2024-04-22T21:29:08.032 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for the mgr to restart... 2024-04-22T21:29:08.032 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr epoch 5... 2024-04-22T21:29:08.144 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:07 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1745617750' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T21:29:12.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:12 smithi083 ceph-mon[21788]: Active manager daemon smithi083.ochghz restarted 2024-04-22T21:29:12.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:12 smithi083 ceph-mon[21788]: Activating manager daemon smithi083.ochghz 2024-04-22T21:29:12.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:12 smithi083 ceph-mon[21788]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T21:29:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-04-22T21:29:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-22T21:29:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "initialized": true 2024-04-22T21:29:13.079 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-04-22T21:29:13.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: mgrmap e6: smithi083.ochghz(active, starting, since 0.052621s) 2024-04-22T21:29:13.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-04-22T21:29:13.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mgr metadata", "who": "smithi083.ochghz", "id": "smithi083.ochghz"}]: dispatch 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: Manager daemon smithi083.ochghz is now available 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: [22/Apr/2024:21:29:12] ENGINE Bus STARTING 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: [22/Apr/2024:21:29:13] ENGINE Serving on https://172.21.15.83:7150 2024-04-22T21:29:13.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: [22/Apr/2024:21:29:13] ENGINE Bus STARTED 2024-04-22T21:29:13.153 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:13.153 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:13 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:29:13.608 INFO:teuthology.orchestra.run.smithi083.stderr:mgr epoch 5 is available 2024-04-22T21:29:13.608 INFO:teuthology.orchestra.run.smithi083.stderr:Setting orchestrator backend to cephadm... 2024-04-22T21:29:14.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:14 smithi083 ceph-mon[21788]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T21:29:14.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:14 smithi083 ceph-mon[21788]: mgrmap e7: smithi083.ochghz(active, since 1.06s) 2024-04-22T21:29:14.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:14 smithi083 ceph-mon[21788]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T21:29:15.280 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: value unchanged 2024-04-22T21:29:15.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:15 smithi083 ceph-mon[21788]: mgrmap e8: smithi083.ochghz(active, since 2s) 2024-04-22T21:29:15.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:15 smithi083 ceph-mon[21788]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:15.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:15 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:15.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:15 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:29:15.853 INFO:teuthology.orchestra.run.smithi083.stderr:Generating ssh key... 2024-04-22T21:29:16.296 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:16 smithi083 ceph-mon[21788]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:17.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:17 smithi083 ceph-mon[21788]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:17.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:17 smithi083 ceph-mon[21788]: Generating ssh key... 2024-04-22T21:29:17.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:17 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:17.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:17 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:18.351 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuL4d6ZMsKhwVDbHQaB8KWFAwc8nIg4m2C9aYBwRGpm9zB0MJ+mjMu6dcZgCVeYxJgqjFEM5r3tnBukXtoZ1xhywSDc1jlbpwuCb29ck78wX8fzOziOMHqlni07LA2fwWbI1BpXaTmnT2kt3mOtHRErWCXxJ9ip6RpXVbKbP/Fn/FplCQqX0N54MC5IOWSVI+o3P4RoERWQ1jS4Q4KMxhb0ky1F32OQpqM5Dv14LJ/vPB60gMUv3lGE8jcu2vioCKZbjyris293DNGJkaNw4unWdyaeMkXOBvcIri/69bm4yAIs1++d0lWQt3J/HZpPJMi4ERipqXuzrY5M8xysUjza6g0tpWGzb6nA6O76JUkKRFovV2kOqIwLr+uQ/ppzKmrB+pG7nyNXg7m8s8sf7EMbTQdrgpGgaazX4X6oVsQCFgtxjx/5bSYbM+H722GduKeQBwxQWxUvhyxqufokrbJ1gE14peXJ9Vr8iLPY5ENZXnSGLfqqXN96b2weIvQCqM= ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:29:18.943 INFO:teuthology.orchestra.run.smithi083.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T21:29:18.943 INFO:teuthology.orchestra.run.smithi083.stderr:Adding key to root@localhost authorized_keys... 2024-04-22T21:29:18.944 INFO:teuthology.orchestra.run.smithi083.stderr:Adding host smithi083... 2024-04-22T21:29:19.199 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:19 smithi083 ceph-mon[21788]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:20.363 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:20 smithi083 ceph-mon[21788]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi083", "addr": "172.21.15.83", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:20.363 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:20 smithi083 ceph-mon[21788]: Deploying cephadm binary to smithi083 2024-04-22T21:29:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Added host 'smithi083' with addr '172.21.15.83' 2024-04-22T21:29:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying mon service with default placement... 2024-04-22T21:29:21.493 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-22T21:29:21.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:21 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:21.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:21 smithi083 ceph-mon[21788]: Added host smithi083 2024-04-22T21:29:21.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:21 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:29:22.015 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying mgr service with default placement... 2024-04-22T21:29:22.655 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:22 smithi083 ceph-mon[21788]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:22.655 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:22 smithi083 ceph-mon[21788]: Saving service mon spec with placement count:5 2024-04-22T21:29:22.655 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:22 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:22.858 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-22T21:29:23.866 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:23 smithi083 ceph-mon[21788]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:23.866 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:23 smithi083 ceph-mon[21788]: Saving service mgr spec with placement count:2 2024-04-22T21:29:23.866 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:23 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:24.313 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying crash service with default placement... 2024-04-22T21:29:25.009 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled crash update... 2024-04-22T21:29:25.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:24 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:25.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:24 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:25.703 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying prometheus service with default placement... 2024-04-22T21:29:26.287 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:26 smithi083 ceph-mon[21788]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:26.288 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:26 smithi083 ceph-mon[21788]: Saving service crash spec with placement * 2024-04-22T21:29:26.288 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:26 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:26.742 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-04-22T21:29:27.263 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying grafana service with default placement... 2024-04-22T21:29:27.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:27 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:27.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:27 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:27.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:27 smithi083 ceph-mon[21788]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:27.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:27 smithi083 ceph-mon[21788]: Saving service prometheus spec with placement count:1 2024-04-22T21:29:27.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:27 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-04-22T21:29:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying node-exporter service with default placement... 2024-04-22T21:29:30.037 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:29 smithi083 ceph-mon[21788]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:30.037 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:29 smithi083 ceph-mon[21788]: Saving service grafana spec with placement count:1 2024-04-22T21:29:30.037 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:29 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:30.038 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-04-22T21:29:30.662 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying alertmanager service with default placement... 2024-04-22T21:29:31.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:31 smithi083 ceph-mon[21788]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:31.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:31 smithi083 ceph-mon[21788]: Saving service node-exporter spec with placement * 2024-04-22T21:29:31.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:31 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:31.310 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-04-22T21:29:32.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:32 smithi083 ceph-mon[21788]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:32.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:32 smithi083 ceph-mon[21788]: Saving service alertmanager spec with placement count:1 2024-04-22T21:29:32.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:32 smithi083 ceph-mon[21788]: from='mgr.14116 172.21.15.83:0/1169665907' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:33.835 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:33 smithi083 ceph-mon[21788]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:29:33.835 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:33 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/680052953' entity='client.admin' 2024-04-22T21:29:34.491 INFO:teuthology.orchestra.run.smithi083.stderr:Enabling the dashboard module... 2024-04-22T21:29:35.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:34 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1759767197' entity='client.admin' 2024-04-22T21:29:36.117 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:35 smithi083 ceph-mon[21788]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:29:36.117 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:35 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/155840029' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T21:29:37.125 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:36 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/155840029' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T21:29:37.126 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:36 smithi083 ceph-mon[21788]: mgrmap e9: smithi083.ochghz(active, since 23s) 2024-04-22T21:29:37.126 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-04-22T21:29:37.127 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-22T21:29:37.127 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": true, 2024-04-22T21:29:37.127 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "active_name": "smithi083.ochghz", 2024-04-22T21:29:37.127 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-22T21:29:37.127 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-04-22T21:29:37.718 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for the mgr to restart... 2024-04-22T21:29:37.718 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr epoch 9... 2024-04-22T21:29:37.972 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:37 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/539834179' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T21:29:42.944 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-04-22T21:29:42.945 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-22T21:29:42.945 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "initialized": true 2024-04-22T21:29:42.945 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-04-22T21:29:43.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: Active manager daemon smithi083.ochghz restarted 2024-04-22T21:29:43.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: Activating manager daemon smithi083.ochghz 2024-04-22T21:29:43.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T21:29:43.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: mgrmap e10: smithi083.ochghz(active, starting, since 0.0531104s) 2024-04-22T21:29:43.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-04-22T21:29:43.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mgr metadata", "who": "smithi083.ochghz", "id": "smithi083.ochghz"}]: dispatch 2024-04-22T21:29:43.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:29:43.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:29:43.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:29:43.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: Manager daemon smithi083.ochghz is now available 2024-04-22T21:29:43.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:29:43.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:29:43.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:29:43.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:29:43.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: [22/Apr/2024:21:29:42] ENGINE Bus STARTING 2024-04-22T21:29:43.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: [22/Apr/2024:21:29:42] ENGINE Serving on https://172.21.15.83:7150 2024-04-22T21:29:43.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: [22/Apr/2024:21:29:42] ENGINE Bus STARTED 2024-04-22T21:29:43.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:42 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:43.515 INFO:teuthology.orchestra.run.smithi083.stderr:mgr epoch 9 is available 2024-04-22T21:29:43.516 INFO:teuthology.orchestra.run.smithi083.stderr:Generating a dashboard self-signed certificate... 2024-04-22T21:29:44.069 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:43 smithi083 ceph-mon[21788]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T21:29:44.069 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:43 smithi083 ceph-mon[21788]: mgrmap e11: smithi083.ochghz(active, since 1.05658s) 2024-04-22T21:29:44.069 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:43 smithi083 ceph-mon[21788]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T21:29:44.069 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:43 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:44.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-22T21:29:44.772 INFO:teuthology.orchestra.run.smithi083.stderr:Creating initial admin user... 2024-04-22T21:29:45.281 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:45 smithi083 ceph-mon[21788]: mgrmap e12: smithi083.ochghz(active, since 2s) 2024-04-22T21:29:45.282 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:45 smithi083 ceph-mon[21788]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:45.282 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:45 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:45.282 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:45 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:45.282 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:45 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$UEV.Ykw/i7G33FB9e.T32e28iu8B4NWGcY3YSFAMLQNpDRyR4yAjG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713821385, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T21:29:46.366 INFO:teuthology.orchestra.run.smithi083.stderr:Fetching dashboard port number... 2024-04-22T21:29:46.968 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:46 smithi083 ceph-mon[21788]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:46.969 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:46 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:47.372 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 8443 2024-04-22T21:29:48.016 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:47 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3399857976' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T21:29:48.028 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-04-22T21:29:48.035 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-04-22T21:29:48.036 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-04-22T21:29:48.036 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T21:29:48.037 INFO:teuthology.orchestra.run.smithi083.stderr:Ceph Dashboard is now available at: 2024-04-22T21:29:48.038 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-22T21:29:48.038 INFO:teuthology.orchestra.run.smithi083.stderr: URL: https://smithi083:8443/ 2024-04-22T21:29:48.038 INFO:teuthology.orchestra.run.smithi083.stderr: User: admin 2024-04-22T21:29:48.038 INFO:teuthology.orchestra.run.smithi083.stderr: Password: 7ev796v1mr 2024-04-22T21:29:48.038 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-22T21:29:48.039 INFO:teuthology.orchestra.run.smithi083.stderr:Enabling autotune for osd_memory_target 2024-04-22T21:29:49.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:49 smithi083 ceph-mon[21788]: mgrmap e13: smithi083.ochghz(active, since 6s) 2024-04-22T21:29:49.982 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-22T21:29:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:You can access the Ceph CLI with: 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr: ceph telemetry on 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:For more information see: 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-22T21:29:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:Bootstrap complete. 2024-04-22T21:29:50.624 INFO:tasks.cephadm:Fetching config... 2024-04-22T21:29:50.625 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-22T21:29:50.625 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T21:29:50.640 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T21:29:50.640 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-22T21:29:50.641 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T21:29:50.696 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T21:29:50.696 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-22T21:29:50.696 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/keyring of=/dev/stdout 2024-04-22T21:29:50.760 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T21:29:50.760 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-22T21:29:50.760 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T21:29:50.815 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T21:29:50.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuL4d6ZMsKhwVDbHQaB8KWFAwc8nIg4m2C9aYBwRGpm9zB0MJ+mjMu6dcZgCVeYxJgqjFEM5r3tnBukXtoZ1xhywSDc1jlbpwuCb29ck78wX8fzOziOMHqlni07LA2fwWbI1BpXaTmnT2kt3mOtHRErWCXxJ9ip6RpXVbKbP/Fn/FplCQqX0N54MC5IOWSVI+o3P4RoERWQ1jS4Q4KMxhb0ky1F32OQpqM5Dv14LJ/vPB60gMUv3lGE8jcu2vioCKZbjyris293DNGJkaNw4unWdyaeMkXOBvcIri/69bm4yAIs1++d0lWQt3J/HZpPJMi4ERipqXuzrY5M8xysUjza6g0tpWGzb6nA6O76JUkKRFovV2kOqIwLr+uQ/ppzKmrB+pG7nyNXg7m8s8sf7EMbTQdrgpGgaazX4X6oVsQCFgtxjx/5bSYbM+H722GduKeQBwxQWxUvhyxqufokrbJ1gE14peXJ9Vr8iLPY5ENZXnSGLfqqXN96b2weIvQCqM= ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T21:29:50.887 INFO:teuthology.orchestra.run.smithi083.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuL4d6ZMsKhwVDbHQaB8KWFAwc8nIg4m2C9aYBwRGpm9zB0MJ+mjMu6dcZgCVeYxJgqjFEM5r3tnBukXtoZ1xhywSDc1jlbpwuCb29ck78wX8fzOziOMHqlni07LA2fwWbI1BpXaTmnT2kt3mOtHRErWCXxJ9ip6RpXVbKbP/Fn/FplCQqX0N54MC5IOWSVI+o3P4RoERWQ1jS4Q4KMxhb0ky1F32OQpqM5Dv14LJ/vPB60gMUv3lGE8jcu2vioCKZbjyris293DNGJkaNw4unWdyaeMkXOBvcIri/69bm4yAIs1++d0lWQt3J/HZpPJMi4ERipqXuzrY5M8xysUjza6g0tpWGzb6nA6O76JUkKRFovV2kOqIwLr+uQ/ppzKmrB+pG7nyNXg7m8s8sf7EMbTQdrgpGgaazX4X6oVsQCFgtxjx/5bSYbM+H722GduKeQBwxQWxUvhyxqufokrbJ1gE14peXJ9Vr8iLPY5ENZXnSGLfqqXN96b2weIvQCqM= ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:29:50.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuL4d6ZMsKhwVDbHQaB8KWFAwc8nIg4m2C9aYBwRGpm9zB0MJ+mjMu6dcZgCVeYxJgqjFEM5r3tnBukXtoZ1xhywSDc1jlbpwuCb29ck78wX8fzOziOMHqlni07LA2fwWbI1BpXaTmnT2kt3mOtHRErWCXxJ9ip6RpXVbKbP/Fn/FplCQqX0N54MC5IOWSVI+o3P4RoERWQ1jS4Q4KMxhb0ky1F32OQpqM5Dv14LJ/vPB60gMUv3lGE8jcu2vioCKZbjyris293DNGJkaNw4unWdyaeMkXOBvcIri/69bm4yAIs1++d0lWQt3J/HZpPJMi4ERipqXuzrY5M8xysUjza6g0tpWGzb6nA6O76JUkKRFovV2kOqIwLr+uQ/ppzKmrB+pG7nyNXg7m8s8sf7EMbTQdrgpGgaazX4X6oVsQCFgtxjx/5bSYbM+H722GduKeQBwxQWxUvhyxqufokrbJ1gE14peXJ9Vr8iLPY5ENZXnSGLfqqXN96b2weIvQCqM= ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T21:29:50.942 INFO:teuthology.orchestra.run.smithi172.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuL4d6ZMsKhwVDbHQaB8KWFAwc8nIg4m2C9aYBwRGpm9zB0MJ+mjMu6dcZgCVeYxJgqjFEM5r3tnBukXtoZ1xhywSDc1jlbpwuCb29ck78wX8fzOziOMHqlni07LA2fwWbI1BpXaTmnT2kt3mOtHRErWCXxJ9ip6RpXVbKbP/Fn/FplCQqX0N54MC5IOWSVI+o3P4RoERWQ1jS4Q4KMxhb0ky1F32OQpqM5Dv14LJ/vPB60gMUv3lGE8jcu2vioCKZbjyris293DNGJkaNw4unWdyaeMkXOBvcIri/69bm4yAIs1++d0lWQt3J/HZpPJMi4ERipqXuzrY5M8xysUjza6g0tpWGzb6nA6O76JUkKRFovV2kOqIwLr+uQ/ppzKmrB+pG7nyNXg7m8s8sf7EMbTQdrgpGgaazX4X6oVsQCFgtxjx/5bSYbM+H722GduKeQBwxQWxUvhyxqufokrbJ1gE14peXJ9Vr8iLPY5ENZXnSGLfqqXN96b2weIvQCqM= ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:29:50.954 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T21:29:51.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:50 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/2701495054' entity='client.admin' 2024-04-22T21:29:55.307 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T21:29:55.308 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T21:29:55.618 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:55 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:55.618 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:55 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi083", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:29:55.618 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:55 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:55.618 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:55 smithi083 ceph-mon[21788]: Deploying daemon alertmanager.smithi083 on smithi083 2024-04-22T21:29:55.618 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:55 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/2943661171' entity='client.admin' 2024-04-22T21:29:58.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:58 smithi083 ceph-mon[21788]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:29:58.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:29:58 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:29:58.989 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi172 2024-04-22T21:29:58.989 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T21:29:58.989 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/etc/ceph/ceph.conf 2024-04-22T21:29:59.007 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T21:29:59.008 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:29:59.065 INFO:tasks.cephadm:Adding host smithi172 to orchestrator... 2024-04-22T21:29:59.065 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch host add smithi172 2024-04-22T21:30:03.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:03 smithi083 ceph-mon[21788]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:04.084 INFO:teuthology.orchestra.run.smithi083.stdout:Added host 'smithi172' with addr '172.21.15.172' 2024-04-22T21:30:05.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:04 smithi083 ceph-mon[21788]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:30:05.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:04 smithi083 ceph-mon[21788]: Deploying cephadm binary to smithi172 2024-04-22T21:30:05.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:04 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:05.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:04 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi083", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:30:05.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:04 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi083", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T21:30:05.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:04 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:30:05.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:04 smithi083 ceph-mon[21788]: Deploying daemon crash.smithi083 on smithi083 2024-04-22T21:30:05.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:04 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:05.679 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T21:30:05.882 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:05 smithi083 ceph-mon[21788]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:05.882 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:05 smithi083 ceph-mon[21788]: Added host smithi172 2024-04-22T21:30:07.545 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:07 smithi083 ceph-mon[21788]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:07.545 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:07 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:07.545 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:07 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:07.545 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:07 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:07.545 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:07 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:30:07.545 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:07 smithi083 ceph-mon[21788]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:30:07.545 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:07 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:07.545 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:07 smithi083 ceph-mon[21788]: Deploying daemon grafana.smithi083 on smithi083 2024-04-22T21:30:07.545 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:07 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:07.749 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:30:07.749 INFO:teuthology.orchestra.run.smithi083.stdout:[{"addr": "172.21.15.83", "hostname": "smithi083", "labels": [], "status": ""}, {"addr": "172.21.15.172", "hostname": "smithi172", "labels": [], "status": ""}] 2024-04-22T21:30:08.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:08 smithi083 ceph-mon[21788]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:30:08.784 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T21:30:08.784 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T21:30:09.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:09 smithi083 ceph-mon[21788]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:11.239 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2024-04-22T21:30:11.557 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:11 smithi083 ceph-mon[21788]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:11.557 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:11 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3785096680' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T21:30:11.909 INFO:tasks.cephadm:Adding mon.smithi083 on smithi083 2024-04-22T21:30:11.909 INFO:tasks.cephadm:Adding mon.smithi172 on smithi172 2024-04-22T21:30:11.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch apply mon '2;smithi083:172.21.15.83=smithi083;smithi172:172.21.15.172=smithi172' 2024-04-22T21:30:12.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:12 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3785096680' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T21:30:12.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:12 smithi083 ceph-mon[21788]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:30:13.633 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:13 smithi083 ceph-mon[21788]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:13.745 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled mon update... 2024-04-22T21:30:14.484 DEBUG:teuthology.orchestra.run.smithi172:mon.smithi172> sudo journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mon.smithi172.service 2024-04-22T21:30:14.486 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:30:14.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:30:15.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:14 smithi083 ceph-mon[21788]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi083:172.21.15.83=smithi083;smithi172:172.21.15.172=smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:30:15.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:14 smithi083 ceph-mon[21788]: Saving service mon spec with placement smithi083:172.21.15.83=smithi083;smithi172:172.21.15.172=smithi172;count:2 2024-04-22T21:30:15.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:14 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:16.051 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:15 smithi083 ceph-mon[21788]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:16.184 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:30:16.184 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:28:45.807818Z","created":"2024-04-22T21:28:45.807818Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:30:16.186 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-04-22T21:30:17.071 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:16 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/3420344988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:30:17.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:30:17.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:30:18.002 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:17 smithi083 ceph-mon[21788]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:19.572 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:30:19.572 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:28:45.807818Z","created":"2024-04-22T21:28:45.807818Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:30:19.574 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-04-22T21:30:19.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:19 smithi083 ceph-mon[21788]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:19.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:19 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/2566150242' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:30:21.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:30:21.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:30:21.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:20 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:21.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:20 smithi083 ceph-mon[21788]: Deploying daemon node-exporter.smithi083 on smithi083 2024-04-22T21:30:22.093 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:21 smithi083 ceph-mon[21788]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:22.896 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:30:22.896 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:28:45.807818Z","created":"2024-04-22T21:28:45.807818Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:30:22.898 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-04-22T21:30:23.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:22 smithi083 ceph-mon[21788]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:23.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:22 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:23.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:22 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:23.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:22 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T21:30:23.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:22 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/1490598651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:30:24.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:30:24.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:30:24.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:24 smithi083 ceph-mon[21788]: from='mgr.14158 172.21.15.83:0/1972721883' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T21:30:24.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:24 smithi083 ceph-mon[21788]: mgrmap e14: smithi083.ochghz(active, since 41s) 2024-04-22T21:30:26.217 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:30:26.217 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:28:45.807818Z","created":"2024-04-22T21:28:45.807818Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:30:26.219 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-04-22T21:30:26.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:26 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/3060950431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:30:27.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:30:27.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:30:29.467 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:30:29.468 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:28:45.807818Z","created":"2024-04-22T21:28:45.807818Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:30:29.469 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-04-22T21:30:29.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:29 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/2252115260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:30:30.825 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:30 smithi083 ceph-mon[21788]: Active manager daemon smithi083.ochghz restarted 2024-04-22T21:30:30.825 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:30 smithi083 ceph-mon[21788]: Activating manager daemon smithi083.ochghz 2024-04-22T21:30:30.825 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:30 smithi083 ceph-mon[21788]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T21:30:30.973 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:30:30.973 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:30:31.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: mgrmap e15: smithi083.ochghz(active, starting, since 0.993775s) 2024-04-22T21:30:31.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-04-22T21:30:31.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mgr metadata", "who": "smithi083.ochghz", "id": "smithi083.ochghz"}]: dispatch 2024-04-22T21:30:31.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:30:31.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:30:31.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:30:31.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: Manager daemon smithi083.ochghz is now available 2024-04-22T21:30:31.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:30:31.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:30:31.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:30:31.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:30:31.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:31.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:30:31.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:30:31.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:31.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:32.775 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:30:32.776 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:28:45.807818Z","created":"2024-04-22T21:28:45.807818Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:30:32.778 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-04-22T21:30:32.785 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:32 smithi083 ceph-mon[21788]: [22/Apr/2024:21:30:31] ENGINE Bus STARTING 2024-04-22T21:30:32.785 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:32 smithi083 ceph-mon[21788]: [22/Apr/2024:21:30:31] ENGINE Serving on https://172.21.15.83:7150 2024-04-22T21:30:32.786 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:32 smithi083 ceph-mon[21788]: [22/Apr/2024:21:30:31] ENGINE Bus STARTED 2024-04-22T21:30:32.786 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:32 smithi083 ceph-mon[21788]: mgrmap e16: smithi083.ochghz(active, since 2s) 2024-04-22T21:30:33.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:33 smithi083 ceph-mon[21788]: mgrmap e17: smithi083.ochghz(active, since 3s) 2024-04-22T21:30:33.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:33 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:33.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:33 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/2832811377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:30:33.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:33 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:34.418 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:30:34.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:30:35.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:34 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:35.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:34 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:35.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:34 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi083", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:30:35.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:34 smithi083 ceph-mon[21788]: Updating smithi083:/etc/ceph/ceph.conf 2024-04-22T21:30:35.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:34 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:36.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:35 smithi083 ceph-mon[21788]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:30:36.780 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:30:36.780 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:28:45.807818Z","created":"2024-04-22T21:28:45.807818Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:30:36.782 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-04-22T21:30:37.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:36 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/122469917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:30:38.383 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:30:38.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:30:38.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:38.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:38.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi172", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:30:38.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:38.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi172", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:30:38.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi172", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T21:30:38.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:30:38.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:38 smithi083 ceph-mon[21788]: Deploying daemon crash.smithi172 on smithi172 2024-04-22T21:30:40.323 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:30:40.324 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:28:45.807818Z","created":"2024-04-22T21:28:45.807818Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:30:40.325 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-04-22T21:30:40.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:40 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/643561574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:30:41.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:30:41.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:30:42.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:42.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi172.edmjti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:30:42.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi172.edmjti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T21:30:42.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:30:42.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:30:42.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:42 smithi083 ceph-mon[21788]: Deploying daemon mgr.smithi172.edmjti on smithi172 2024-04-22T21:30:44.132 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:30:44.132 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:28:45.807818Z","created":"2024-04-22T21:28:45.807818Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:30:44.134 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-04-22T21:30:44.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:44 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/4284906355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:30:45.944 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:30:45.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:30:46.453 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:46 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:46.453 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:46 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:30:46.453 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:46 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:30:46.453 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:46 smithi083 ceph-mon[21788]: Deploying daemon mon.smithi172 on smithi172 2024-04-22T21:30:46.453 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:46 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:49.046 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:30:49.046 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:28:45.807818Z","created":"2024-04-22T21:28:45.807818Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:30:49.047 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-04-22T21:30:49.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:49 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/1825224032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:30:51.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T21:30:51.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:30:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-04-22T21:30:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: mon.smithi083 calling monitor election 2024-04-22T21:30:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi172"}]: dispatch 2024-04-22T21:30:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi172"}]: dispatch 2024-04-22T21:30:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.? 172.21.15.172:0/296208332' entity='mgr.smithi172.edmjti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi172.edmjti/crt"}]: dispatch 2024-04-22T21:30:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi172"}]: dispatch 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: mon.smithi172 calling monitor election 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi172"}]: dispatch 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi172"}]: dispatch 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi172"}]: dispatch 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: mon.smithi083 is new leader, mons smithi083,smithi172 in quorum (ranks 0,1) 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: monmap e2: 2 mons at {smithi083=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],smithi172=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0]} 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: fsmap 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: mgrmap e17: smithi083.ochghz(active, since 25s) 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: Standby manager daemon smithi172.edmjti started 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.? 172.21.15.172:0/296208332' entity='mgr.smithi172.edmjti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: overall HEALTH_OK 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.? 172.21.15.172:0/296208332' entity='mgr.smithi172.edmjti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi172.edmjti/key"}]: dispatch 2024-04-22T21:30:55.152 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:55.153 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:54 smithi083 ceph-mon[21788]: from='mgr.? 172.21.15.172:0/296208332' entity='mgr.smithi172.edmjti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T21:30:55.894 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T21:30:55.894 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":2,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:30:49.615660Z","created":"2024-04-22T21:28:45.807818Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi172","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T21:30:55.895 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 2 2024-04-22T21:30:56.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:55 smithi083 ceph-mon[21788]: Deploying daemon node-exporter.smithi172 on smithi172 2024-04-22T21:30:56.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:55 smithi083 ceph-mon[21788]: mgrmap e18: smithi083.ochghz(active, since 25s), standbys: smithi172.edmjti 2024-04-22T21:30:56.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mgr metadata", "who": "smithi172.edmjti", "id": "smithi172.edmjti"}]: dispatch 2024-04-22T21:30:56.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:56.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi172"}]: dispatch 2024-04-22T21:30:56.794 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T21:30:56.794 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T21:30:57.085 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:56 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/658388790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:30:57.085 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:56 smithi083 ceph-mon[21788]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:58.673 INFO:teuthology.orchestra.run.smithi083.stdout:# minimal ceph.conf for 3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:30:58.674 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-04-22T21:30:58.674 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 3a5a1c14-00ef-11ef-bc93-c7b262605968 2024-04-22T21:30:58.674 INFO:teuthology.orchestra.run.smithi083.stdout: mon_host = [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] 2024-04-22T21:30:59.276 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T21:30:59.276 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-22T21:30:59.276 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T21:30:59.302 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-22T21:30:59.302 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:30:59.368 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T21:30:59.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T21:30:59.399 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T21:30:59.399 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:30:59.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:59 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:30:59.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:59 smithi083 ceph-mon[21788]: Deploying daemon prometheus.smithi083 on smithi083 2024-04-22T21:30:59.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:59 smithi083 ceph-mon[21788]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:30:59.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:30:59 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1220424511' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:30:59.467 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T21:30:59.467 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-22T21:30:59.467 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T21:30:59.482 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T21:30:59.482 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-04-22T21:30:59.537 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T21:30:59.537 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:30:59.537 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T21:30:59.537 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:30:59.537 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:30:59.537 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-04-22 21:30:20.420645945 +0000 2024-04-22T21:30:59.538 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-04-22 21:27:54.237203882 +0000 2024-04-22T21:30:59.538 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-04-22 21:27:54.237203882 +0000 2024-04-22T21:30:59.538 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-04-22 21:27:54.237203882 +0000 2024-04-22T21:30:59.538 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T21:30:59.600 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-04-22T21:30:59.600 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-04-22T21:30:59.600 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000212058 s, 2.4 MB/s 2024-04-22T21:30:59.601 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T21:30:59.657 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-04-22T21:30:59.712 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T21:30:59.712 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:30:59.712 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-22T21:30:59.712 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:30:59.712 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:30:59.713 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-04-22 21:30:20.420645945 +0000 2024-04-22T21:30:59.713 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-04-22 21:27:54.579195052 +0000 2024-04-22T21:30:59.713 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-04-22 21:27:54.579195052 +0000 2024-04-22T21:30:59.713 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-04-22 21:27:54.579195052 +0000 2024-04-22T21:30:59.713 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T21:30:59.776 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-04-22T21:30:59.776 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-04-22T21:30:59.776 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000239405 s, 2.1 MB/s 2024-04-22T21:30:59.777 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T21:30:59.834 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-04-22T21:30:59.888 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T21:30:59.888 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:30:59.888 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-22T21:30:59.888 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:30:59.888 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:30:59.888 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-04-22 21:30:20.420645945 +0000 2024-04-22T21:30:59.889 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-04-22 21:27:54.939185757 +0000 2024-04-22T21:30:59.889 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-04-22 21:27:54.939185757 +0000 2024-04-22T21:30:59.889 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-04-22 21:27:54.939185757 +0000 2024-04-22T21:30:59.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T21:30:59.951 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-04-22T21:30:59.951 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-04-22T21:30:59.951 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000191351 s, 2.7 MB/s 2024-04-22T21:30:59.952 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T21:31:00.008 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-04-22T21:31:00.062 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T21:31:00.062 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:31:00.062 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-22T21:31:00.062 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:31:00.062 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:31:00.062 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-04-22 21:30:20.421645920 +0000 2024-04-22T21:31:00.062 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-04-22 21:27:55.292176643 +0000 2024-04-22T21:31:00.062 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-04-22 21:27:55.292176643 +0000 2024-04-22T21:31:00.062 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-04-22 21:27:55.292176643 +0000 2024-04-22T21:31:00.063 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T21:31:00.123 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-04-22T21:31:00.123 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-04-22T21:31:00.123 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.00017149 s, 3.0 MB/s 2024-04-22T21:31:00.125 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T21:31:00.179 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T21:31:00.179 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T21:31:00.198 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T21:31:00.198 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-04-22T21:31:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T21:31:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:31:00.256 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-22T21:31:00.256 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:31:00.256 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:31:00.256 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-22 21:30:55.206430669 +0000 2024-04-22T21:31:00.256 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-22 21:27:54.236559683 +0000 2024-04-22T21:31:00.257 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-22 21:27:54.236559683 +0000 2024-04-22T21:31:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-22 21:27:54.236559683 +0000 2024-04-22T21:31:00.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T21:31:00.321 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-22T21:31:00.322 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-22T21:31:00.322 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000233178 s, 2.2 MB/s 2024-04-22T21:31:00.323 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T21:31:00.381 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-04-22T21:31:00.439 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T21:31:00.439 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:31:00.439 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T21:31:00.439 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:31:00.439 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:31:00.439 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-22 21:30:55.207430645 +0000 2024-04-22T21:31:00.439 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-22 21:27:54.572551498 +0000 2024-04-22T21:31:00.439 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-22 21:27:54.572551498 +0000 2024-04-22T21:31:00.439 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-22 21:27:54.572551498 +0000 2024-04-22T21:31:00.440 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T21:31:00.506 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-22T21:31:00.507 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-22T21:31:00.507 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000251298 s, 2.0 MB/s 2024-04-22T21:31:00.508 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T21:31:00.567 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-04-22T21:31:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T21:31:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:31:00.625 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-22T21:31:00.625 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:31:00.625 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:31:00.625 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-22 21:30:55.207430645 +0000 2024-04-22T21:31:00.625 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-22 21:27:54.931542753 +0000 2024-04-22T21:31:00.625 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-22 21:27:54.931542753 +0000 2024-04-22T21:31:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-22 21:27:54.931542753 +0000 2024-04-22T21:31:00.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T21:31:00.690 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-22T21:31:00.690 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-22T21:31:00.690 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000212279 s, 2.4 MB/s 2024-04-22T21:31:00.691 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T21:31:00.748 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-04-22T21:31:00.805 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T21:31:00.806 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:31:00.806 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-22T21:31:00.806 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:31:00.806 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:31:00.806 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-22 21:30:55.207430645 +0000 2024-04-22T21:31:00.806 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-22 21:27:55.293533934 +0000 2024-04-22T21:31:00.806 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-22 21:27:55.293533934 +0000 2024-04-22T21:31:00.806 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-22 21:27:55.293533934 +0000 2024-04-22T21:31:00.806 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T21:31:00.870 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-22T21:31:00.870 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-22T21:31:00.870 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000203904 s, 2.5 MB/s 2024-04-22T21:31:00.872 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T21:31:00.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:00 smithi083 ceph-mon[21788]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:00.929 INFO:tasks.cephadm:Deploying osd.0 on smithi083 with /dev/vg_nvme/lv_4... 2024-04-22T21:31:00.929 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T21:31:01.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:01 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:01.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:01 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:02.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:02 smithi172 ceph-mon[23940]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:02.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:02 smithi083 ceph-mon[21788]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:03.864 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T21:31:03.864 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T21:31:03.865 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-04-22T21:31:03.865 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-04-22T21:31:03.865 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131054 s, 800 MB/s 2024-04-22T21:31:03.865 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-04-22T21:31:04.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:04 smithi083 ceph-mon[21788]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:05.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:04 smithi172 ceph-mon[23940]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:05.933 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_4 2024-04-22T21:31:06.887 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:06 smithi083 ceph-mon[21788]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:07.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:06 smithi172 ceph-mon[23940]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:09.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:09 smithi083 ceph-mon[21788]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:09.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:09 smithi172 ceph-mon[23940]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:10.711 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:10 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:31:10.711 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:10 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:31:10.711 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:10 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:10.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:10 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:31:10.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:10 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:31:10.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:10 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:11.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:11 smithi083 ceph-mon[21788]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:31:11.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:11 smithi083 ceph-mon[21788]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:11.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:11 smithi172 ceph-mon[23940]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:31:11.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:11 smithi172 ceph-mon[23940]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:12.852 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:12 smithi083 ceph-mon[21788]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:12.852 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:12 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:12.852 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:12 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:31:12.852 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:12 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:12.852 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:12 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:31:13.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:12 smithi172 ceph-mon[23940]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:13.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:12 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:13.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:12 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:31:13.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:12 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:13.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:12 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:31:14.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:14 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1089467516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "120f52f4-cd77-4920-9b7f-9e02e652e3e8"}]: dispatch 2024-04-22T21:31:14.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:14 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1089467516' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "120f52f4-cd77-4920-9b7f-9e02e652e3e8"}]': finished 2024-04-22T21:31:14.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:14 smithi083 ceph-mon[21788]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T21:31:14.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:14 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:14.414 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:14 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/1089467516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "120f52f4-cd77-4920-9b7f-9e02e652e3e8"}]: dispatch 2024-04-22T21:31:14.415 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:14 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/1089467516' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "120f52f4-cd77-4920-9b7f-9e02e652e3e8"}]': finished 2024-04-22T21:31:14.415 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:14 smithi172 ceph-mon[23940]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T21:31:14.415 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:14 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:15.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:15 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:15.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:15 smithi083 ceph-mon[21788]: Updating smithi172:/etc/ceph/ceph.conf 2024-04-22T21:31:15.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:15 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/2813762536' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:31:15.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:15 smithi083 ceph-mon[21788]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:31:15.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:15 smithi083 ceph-mon[21788]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:15.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:15 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:15.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:15 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:15.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:15 smithi172 ceph-mon[23940]: Updating smithi172:/etc/ceph/ceph.conf 2024-04-22T21:31:15.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:15 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/2813762536' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:31:15.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:15 smithi172 ceph-mon[23940]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:31:15.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:15 smithi172 ceph-mon[23940]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:15.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:15 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:16.779 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:16.779 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:16 smithi083 ceph-mon[21788]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:16.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:16.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:16 smithi172 ceph-mon[23940]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:17.985 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:17.985 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:17 smithi083 ceph-mon[21788]: Updating smithi083:/etc/ceph/ceph.conf 2024-04-22T21:31:17.985 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:17.986 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:17.986 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:17 smithi083 ceph-mon[21788]: Reconfiguring mon.smithi083 (unknown last config time)... 2024-04-22T21:31:17.986 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:31:17.986 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:31:17.986 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:17.986 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:17 smithi083 ceph-mon[21788]: Reconfiguring daemon mon.smithi083 on smithi083 2024-04-22T21:31:18.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:18.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:17 smithi172 ceph-mon[23940]: Updating smithi083:/etc/ceph/ceph.conf 2024-04-22T21:31:18.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:18.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:18.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:17 smithi172 ceph-mon[23940]: Reconfiguring mon.smithi083 (unknown last config time)... 2024-04-22T21:31:18.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:31:18.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:31:18.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:18.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:17 smithi172 ceph-mon[23940]: Reconfiguring daemon mon.smithi083 on smithi083 2024-04-22T21:31:19.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:19 smithi083 ceph-mon[21788]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:19.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:19 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:19.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:19 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi083.ochghz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:31:19.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:19 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:31:19.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:19 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:19.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:19 smithi172 ceph-mon[23940]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:19.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:19 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:19.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:19 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi083.ochghz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:31:19.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:19 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:31:19.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:19 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:20.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:20 smithi083 ceph-mon[21788]: Reconfiguring mgr.smithi083.ochghz (unknown last config time)... 2024-04-22T21:31:20.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:20 smithi083 ceph-mon[21788]: Reconfiguring daemon mgr.smithi083.ochghz on smithi083 2024-04-22T21:31:20.472 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:20 smithi172 ceph-mon[23940]: Reconfiguring mgr.smithi083.ochghz (unknown last config time)... 2024-04-22T21:31:20.472 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:20 smithi172 ceph-mon[23940]: Reconfiguring daemon mgr.smithi083.ochghz on smithi083 2024-04-22T21:31:21.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:21 smithi083 ceph-mon[21788]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:21.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:21.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:21 smithi172 ceph-mon[23940]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:21.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:22.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:22 smithi172 ceph-mon[23940]: Reconfiguring alertmanager.smithi083 (dependencies changed)... 2024-04-22T21:31:22.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:22 smithi172 ceph-mon[23940]: Reconfiguring daemon alertmanager.smithi083 on smithi083 2024-04-22T21:31:22.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:22 smithi083 ceph-mon[21788]: Reconfiguring alertmanager.smithi083 (dependencies changed)... 2024-04-22T21:31:22.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:22 smithi083 ceph-mon[21788]: Reconfiguring daemon alertmanager.smithi083 on smithi083 2024-04-22T21:31:23.230 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:23 smithi083 ceph-mon[21788]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:23.230 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:23 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T21:31:23.230 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:23 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:23.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:23 smithi172 ceph-mon[23940]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:23.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:23 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T21:31:23.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:23 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:24.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:24 smithi083 ceph-mon[21788]: Deploying daemon osd.0 on smithi083 2024-04-22T21:31:24.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:24 smithi172 ceph-mon[23940]: Deploying daemon osd.0 on smithi083 2024-04-22T21:31:25.227 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:25 smithi083 ceph-mon[21788]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:25.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:25 smithi172 ceph-mon[23940]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:26.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:26 smithi083 ceph-mon[21788]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:27.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:26 smithi172 ceph-mon[23940]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:28.125 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:28 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:28.125 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:28 smithi083 ceph-mon[21788]: Reconfiguring crash.smithi083 (monmap changed)... 2024-04-22T21:31:28.125 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:28 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi083", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:31:28.125 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:28 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:28.125 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:28 smithi083 ceph-mon[21788]: Reconfiguring daemon crash.smithi083 on smithi083 2024-04-22T21:31:28.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:28 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:28.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:28 smithi172 ceph-mon[23940]: Reconfiguring crash.smithi083 (monmap changed)... 2024-04-22T21:31:28.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:28 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi083", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:31:28.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:28 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:28.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:28 smithi172 ceph-mon[23940]: Reconfiguring daemon crash.smithi083 on smithi083 2024-04-22T21:31:29.390 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:29 smithi083 ceph-mon[21788]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:29.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:29 smithi172 ceph-mon[23940]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:30.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:30 smithi172 ceph-mon[23940]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:30.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:31:30.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:31:30.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:30 smithi083 ceph-mon[21788]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:30.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:31:30.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:31:33.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:33 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:33.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:33 smithi172 ceph-mon[23940]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:33.289 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:33 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:33.289 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:33 smithi083 ceph-mon[21788]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:34.359 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:34 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:34.360 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:34 smithi083 ceph-mon[21788]: Reconfiguring grafana.smithi083 (dependencies changed)... 2024-04-22T21:31:34.360 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:34 smithi083 ceph-mon[21788]: Reconfiguring daemon grafana.smithi083 on smithi083 2024-04-22T21:31:34.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:34 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:34.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:34 smithi172 ceph-mon[23940]: Reconfiguring grafana.smithi083 (dependencies changed)... 2024-04-22T21:31:34.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:34 smithi172 ceph-mon[23940]: Reconfiguring daemon grafana.smithi083 on smithi083 2024-04-22T21:31:35.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:35 smithi172 ceph-mon[23940]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:35.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:35 smithi172 ceph-mon[23940]: from='osd.0 [v2:172.21.15.83:6802/2774972645,v1:172.21.15.83:6803/2774972645]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T21:31:35.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:35 smithi083 ceph-mon[21788]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:35.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:35 smithi083 ceph-mon[21788]: from='osd.0 [v2:172.21.15.83:6802/2774972645,v1:172.21.15.83:6803/2774972645]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T21:31:36.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:36 smithi172 ceph-mon[23940]: from='osd.0 [v2:172.21.15.83:6802/2774972645,v1:172.21.15.83:6803/2774972645]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T21:31:36.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:36 smithi172 ceph-mon[23940]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T21:31:36.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:36 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:36.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:36 smithi172 ceph-mon[23940]: from='osd.0 [v2:172.21.15.83:6802/2774972645,v1:172.21.15.83:6803/2774972645]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-04-22T21:31:36.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:36 smithi083 ceph-mon[21788]: from='osd.0 [v2:172.21.15.83:6802/2774972645,v1:172.21.15.83:6803/2774972645]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T21:31:36.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:36 smithi083 ceph-mon[21788]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T21:31:36.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:36 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:36.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:36 smithi083 ceph-mon[21788]: from='osd.0 [v2:172.21.15.83:6802/2774972645,v1:172.21.15.83:6803/2774972645]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-04-22T21:31:37.367 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:37 smithi083 ceph-mon[21788]: from='osd.0 [v2:172.21.15.83:6802/2774972645,v1:172.21.15.83:6803/2774972645]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-04-22T21:31:37.367 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:37 smithi083 ceph-mon[21788]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T21:31:37.368 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:37 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:37.368 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:37 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:37.368 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:37 smithi083 ceph-mon[21788]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:37.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:37 smithi172 ceph-mon[23940]: from='osd.0 [v2:172.21.15.83:6802/2774972645,v1:172.21.15.83:6803/2774972645]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-04-22T21:31:37.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:37 smithi172 ceph-mon[23940]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T21:31:37.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:37 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:37.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:37 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:37.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:37 smithi172 ceph-mon[23940]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:31:38.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:38 smithi172 ceph-mon[23940]: purged_snaps scrub starts 2024-04-22T21:31:38.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:38 smithi172 ceph-mon[23940]: purged_snaps scrub ok 2024-04-22T21:31:38.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:38.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:38 smithi172 ceph-mon[23940]: osd.0 [v2:172.21.15.83:6802/2774972645,v1:172.21.15.83:6803/2774972645] boot 2024-04-22T21:31:38.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:38 smithi172 ceph-mon[23940]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T21:31:38.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:38.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:38.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi172", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:31:38.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:38.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:38 smithi083 ceph-mon[21788]: purged_snaps scrub starts 2024-04-22T21:31:38.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:38 smithi083 ceph-mon[21788]: purged_snaps scrub ok 2024-04-22T21:31:38.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:38.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:38 smithi083 ceph-mon[21788]: osd.0 [v2:172.21.15.83:6802/2774972645,v1:172.21.15.83:6803/2774972645] boot 2024-04-22T21:31:38.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:38 smithi083 ceph-mon[21788]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T21:31:38.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:31:38.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:38.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi172", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T21:31:38.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:39.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:39 smithi172 ceph-mon[23940]: Reconfiguring crash.smithi172 (monmap changed)... 2024-04-22T21:31:39.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:39 smithi172 ceph-mon[23940]: Reconfiguring daemon crash.smithi172 on smithi172 2024-04-22T21:31:39.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:39 smithi172 ceph-mon[23940]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:39.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:39 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:39.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:39 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi172.edmjti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:31:39.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:39 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:31:39.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:39 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:39.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:39 smithi172 ceph-mon[23940]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T21:31:39.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:39 smithi083 ceph-mon[21788]: Reconfiguring crash.smithi172 (monmap changed)... 2024-04-22T21:31:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:39 smithi083 ceph-mon[21788]: Reconfiguring daemon crash.smithi172 on smithi172 2024-04-22T21:31:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:39 smithi083 ceph-mon[21788]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:39 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:39 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi172.edmjti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:31:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:39 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:31:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:39 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:39 smithi083 ceph-mon[21788]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T21:31:40.077 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 0 on host 'smithi083' 2024-04-22T21:31:40.352 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:40 smithi172 ceph-mon[23940]: Reconfiguring mgr.smithi172.edmjti (monmap changed)... 2024-04-22T21:31:40.352 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:40 smithi172 ceph-mon[23940]: Reconfiguring daemon mgr.smithi172.edmjti on smithi172 2024-04-22T21:31:40.352 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:40 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:40.352 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:40 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:40.352 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:40 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:31:40.352 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:40 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:31:40.352 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:40 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:40.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:40 smithi083 ceph-mon[21788]: Reconfiguring mgr.smithi172.edmjti (monmap changed)... 2024-04-22T21:31:40.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:40 smithi083 ceph-mon[21788]: Reconfiguring daemon mgr.smithi172.edmjti on smithi172 2024-04-22T21:31:40.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:40 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:40.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:40 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:40.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:40 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:31:40.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:40 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:31:40.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:40 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:40.697 DEBUG:teuthology.orchestra.run.smithi083:osd.0> sudo journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T21:31:40.700 INFO:tasks.cephadm:Deploying osd.1 on smithi083 with /dev/vg_nvme/lv_3... 2024-04-22T21:31:40.700 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T21:31:41.506 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:41 smithi172 ceph-mon[23940]: Reconfiguring mon.smithi172 (monmap changed)... 2024-04-22T21:31:41.507 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:41 smithi172 ceph-mon[23940]: Reconfiguring daemon mon.smithi172 on smithi172 2024-04-22T21:31:41.507 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:41 smithi172 ceph-mon[23940]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:41.602 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:41 smithi083 ceph-mon[21788]: Reconfiguring mon.smithi172 (monmap changed)... 2024-04-22T21:31:41.602 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:41 smithi083 ceph-mon[21788]: Reconfiguring daemon mon.smithi172 on smithi172 2024-04-22T21:31:41.602 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:41 smithi083 ceph-mon[21788]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:42.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:42.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:31:42.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:31:42.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.83:9093"}]: dispatch 2024-04-22T21:31:42.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.83:9093"}]: dispatch 2024-04-22T21:31:42.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:42.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:31:42.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:31:42.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.83:3000"}]: dispatch 2024-04-22T21:31:42.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.83:3000"}]: dispatch 2024-04-22T21:31:42.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:42.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:31:42.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:31:42.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.83:9095"}]: dispatch 2024-04-22T21:31:42.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.83:9095"}]: dispatch 2024-04-22T21:31:42.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:42.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:31:42.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:42.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:42 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:31:42.791 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:42.791 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:31:42.791 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.83:9093"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.83:9093"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.83:3000"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.83:3000"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.83:9095"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.83:9095"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:31:42.792 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:42.793 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:42 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:31:42.844 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T21:31:42.844 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T21:31:42.844 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-04-22T21:31:42.844 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-04-22T21:31:42.844 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126629 s, 828 MB/s 2024-04-22T21:31:42.844 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-04-22T21:31:43.494 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_3 2024-04-22T21:31:43.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:43 smithi172 ceph-mon[23940]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:43.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:43.879 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:43 smithi083 ceph-mon[21788]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:43.879 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:44.853 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:44 smithi083 ceph-mon[21788]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:45.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:44 smithi172 ceph-mon[23940]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:45.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:45 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:31:45.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:45 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:31:45.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:45 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:46.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:45 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:31:46.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:45 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:31:46.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:45 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:46.991 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:46 smithi083 ceph-mon[21788]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:31:46.992 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:46 smithi083 ceph-mon[21788]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:47.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:46 smithi172 ceph-mon[23940]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:31:47.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:46 smithi172 ceph-mon[23940]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:49.390 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:49 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3650159033' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e885f222-674f-4a38-83bb-ab008b20607b"}]: dispatch 2024-04-22T21:31:49.390 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:49 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3650159033' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e885f222-674f-4a38-83bb-ab008b20607b"}]': finished 2024-04-22T21:31:49.390 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:49 smithi083 ceph-mon[21788]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T21:31:49.390 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:49 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:31:49.390 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:49 smithi083 ceph-mon[21788]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:49.390 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:49 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/4247262870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:31:49.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:49 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/3650159033' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e885f222-674f-4a38-83bb-ab008b20607b"}]: dispatch 2024-04-22T21:31:49.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:49 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/3650159033' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e885f222-674f-4a38-83bb-ab008b20607b"}]': finished 2024-04-22T21:31:49.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:49 smithi172 ceph-mon[23940]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T21:31:49.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:49 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:31:49.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:49 smithi172 ceph-mon[23940]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:49.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:49 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/4247262870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:31:50.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:50.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:50 smithi172 ceph-mon[23940]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:50.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:50.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:50 smithi083 ceph-mon[21788]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:52.749 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:52 smithi083 ceph-mon[21788]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:52.894 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:52 smithi172 ceph-mon[23940]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:55 smithi172 ceph-mon[23940]: Detected new or changed devices on smithi083 2024-04-22T21:31:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:31:55.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:55 smithi172 ceph-mon[23940]: Adjusting osd_memory_target on smithi083 to 12864M 2024-04-22T21:31:55.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:55.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:55.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:31:55.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:55.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:31:55.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:55 smithi172 ceph-mon[23940]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:55 smithi083 ceph-mon[21788]: Detected new or changed devices on smithi083 2024-04-22T21:31:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:31:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:55 smithi083 ceph-mon[21788]: Adjusting osd_memory_target on smithi083 to 12864M 2024-04-22T21:31:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:31:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:31:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:55 smithi083 ceph-mon[21788]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:56.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:56 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T21:31:56.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:56 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:56.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:56 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:56.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:56 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T21:31:56.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:56 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:31:56.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:56 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:31:57.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:57 smithi083 ceph-mon[21788]: Deploying daemon osd.1 on smithi083 2024-04-22T21:31:57.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:57 smithi083 ceph-mon[21788]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:57.597 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:57 smithi172 ceph-mon[23940]: Deploying daemon osd.1 on smithi083 2024-04-22T21:31:57.597 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:57 smithi172 ceph-mon[23940]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:59.330 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:31:59 smithi083 ceph-mon[21788]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:31:59.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:31:59 smithi172 ceph-mon[23940]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:32:00.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:00 smithi172 ceph-mon[23940]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:32:00.897 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:00 smithi083 ceph-mon[21788]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:32:01.886 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:01 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:01.886 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:01 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:02.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:01 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:02.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:01 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:02.787 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:02 smithi083 ceph-mon[21788]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:32:03.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:02 smithi172 ceph-mon[23940]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:32:04.064 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:03 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:04.064 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:03 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:32:04.064 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:03 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:04.065 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:03 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:32:04.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:03 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:04.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:03 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:32:04.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:03 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:04.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:03 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:32:05.767 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:05.767 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:05 smithi083 ceph-mon[21788]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:32:05.797 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 1 on host 'smithi083' 2024-04-22T21:32:05.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:05.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:05 smithi172 ceph-mon[23940]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:32:06.422 DEBUG:teuthology.orchestra.run.smithi083:osd.1> sudo journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T21:32:06.424 INFO:tasks.cephadm:Deploying osd.2 on smithi083 with /dev/vg_nvme/lv_2... 2024-04-22T21:32:06.424 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T21:32:07.022 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:06 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:07.022 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:06 smithi083 ceph-mon[21788]: from='osd.1 [v2:172.21.15.83:6810/2475641293,v1:172.21.15.83:6811/2475641293]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T21:32:07.022 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:06 smithi083 ceph-mon[21788]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:32:07.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:06 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:07.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:06 smithi172 ceph-mon[23940]: from='osd.1 [v2:172.21.15.83:6810/2475641293,v1:172.21.15.83:6811/2475641293]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T21:32:07.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:06 smithi172 ceph-mon[23940]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:32:08.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:07 smithi172 ceph-mon[23940]: from='osd.1 [v2:172.21.15.83:6810/2475641293,v1:172.21.15.83:6811/2475641293]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T21:32:08.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:07 smithi172 ceph-mon[23940]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T21:32:08.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:07 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:32:08.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:07 smithi172 ceph-mon[23940]: from='osd.1 [v2:172.21.15.83:6810/2475641293,v1:172.21.15.83:6811/2475641293]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-04-22T21:32:08.114 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:07 smithi083 ceph-mon[21788]: from='osd.1 [v2:172.21.15.83:6810/2475641293,v1:172.21.15.83:6811/2475641293]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T21:32:08.114 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:07 smithi083 ceph-mon[21788]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T21:32:08.115 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:07 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:32:08.115 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:07 smithi083 ceph-mon[21788]: from='osd.1 [v2:172.21.15.83:6810/2475641293,v1:172.21.15.83:6811/2475641293]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-04-22T21:32:08.115 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:32:07 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1[36173]: 2024-04-22T21:32:07.823+0000 7fce902ff700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T21:32:08.115 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:32:07 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1[36173]: 2024-04-22T21:32:07.826+0000 7fce8c499700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:32:08.962 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:08 smithi083 ceph-mon[21788]: from='osd.1 [v2:172.21.15.83:6810/2475641293,v1:172.21.15.83:6811/2475641293]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-04-22T21:32:08.963 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:08 smithi083 ceph-mon[21788]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T21:32:08.963 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:08 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:32:08.963 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:08 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:32:08.963 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:08 smithi083 ceph-mon[21788]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:32:08.963 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:08 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:32:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T21:32:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T21:32:08.994 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-04-22T21:32:08.995 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-04-22T21:32:08.995 INFO:teuthology.orchestra.run.smithi083.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.014877 s, 705 MB/s 2024-04-22T21:32:08.995 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 2024-04-22T21:32:08.995 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-04-22T21:32:09.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:08 smithi172 ceph-mon[23940]: from='osd.1 [v2:172.21.15.83:6810/2475641293,v1:172.21.15.83:6811/2475641293]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-04-22T21:32:09.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:08 smithi172 ceph-mon[23940]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T21:32:09.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:08 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:32:09.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:08 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:32:09.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:08 smithi172 ceph-mon[23940]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:32:09.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:08 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:32:09.613 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_2 2024-04-22T21:32:09.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:09 smithi083 ceph-mon[21788]: purged_snaps scrub starts 2024-04-22T21:32:09.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:09 smithi083 ceph-mon[21788]: purged_snaps scrub ok 2024-04-22T21:32:09.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:09 smithi083 ceph-mon[21788]: osd.1 [v2:172.21.15.83:6810/2475641293,v1:172.21.15.83:6811/2475641293] boot 2024-04-22T21:32:09.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:09 smithi083 ceph-mon[21788]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T21:32:09.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:09 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:32:10.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:09 smithi172 ceph-mon[23940]: purged_snaps scrub starts 2024-04-22T21:32:10.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:09 smithi172 ceph-mon[23940]: purged_snaps scrub ok 2024-04-22T21:32:10.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:09 smithi172 ceph-mon[23940]: osd.1 [v2:172.21.15.83:6810/2475641293,v1:172.21.15.83:6811/2475641293] boot 2024-04-22T21:32:10.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:09 smithi172 ceph-mon[23940]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T21:32:10.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:09 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:32:10.873 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:10 smithi083 ceph-mon[21788]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T21:32:10.873 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:10 smithi083 ceph-mon[21788]: pgmap v53: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:11.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:10 smithi172 ceph-mon[23940]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T21:32:11.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:10 smithi172 ceph-mon[23940]: pgmap v53: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:12.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:32:12.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:32:12.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:12.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:32:12.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:32:12.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:13.039 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:12 smithi083 ceph-mon[21788]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:32:13.039 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:12 smithi083 ceph-mon[21788]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:13.039 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:12 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:13.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:12 smithi172 ceph-mon[23940]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:32:13.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:12 smithi172 ceph-mon[23940]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:13.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:12 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:14.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:14 smithi083 ceph-mon[21788]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:15.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:14 smithi172 ceph-mon[23940]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:15.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:15 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1896676906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad91da27-115e-4ffc-8885-a6eb7fd412bd"}]: dispatch 2024-04-22T21:32:15.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:15 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1896676906' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad91da27-115e-4ffc-8885-a6eb7fd412bd"}]': finished 2024-04-22T21:32:15.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:15 smithi083 ceph-mon[21788]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T21:32:15.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:15 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:15.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:15 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/826517383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:32:16.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:15 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/1896676906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad91da27-115e-4ffc-8885-a6eb7fd412bd"}]: dispatch 2024-04-22T21:32:16.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:15 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/1896676906' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad91da27-115e-4ffc-8885-a6eb7fd412bd"}]': finished 2024-04-22T21:32:16.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:15 smithi172 ceph-mon[23940]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T21:32:16.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:15 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:16.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:15 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/826517383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:32:16.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:16 smithi083 ceph-mon[21788]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:17.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:16 smithi172 ceph-mon[23940]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:19.388 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:19 smithi083 ceph-mon[21788]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:19.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:19 smithi172 ceph-mon[23940]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:21.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:21 smithi172 ceph-mon[23940]: Detected new or changed devices on smithi083 2024-04-22T21:32:21.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:21.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:32:21.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:32:21.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:21 smithi172 ceph-mon[23940]: Adjusting osd_memory_target on smithi083 to 6432M 2024-04-22T21:32:21.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:21.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:21.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:32:21.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:21.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:32:21.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:21 smithi172 ceph-mon[23940]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:21.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:21 smithi083 ceph-mon[21788]: Detected new or changed devices on smithi083 2024-04-22T21:32:21.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:21.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:32:21.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:32:21.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:21 smithi083 ceph-mon[21788]: Adjusting osd_memory_target on smithi083 to 6432M 2024-04-22T21:32:21.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:21.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:21.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:32:21.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:21.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:32:21.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:21 smithi083 ceph-mon[21788]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:22.555 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:22 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T21:32:22.555 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:22 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:22.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:22 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T21:32:22.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:22 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:23.569 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:23 smithi083 ceph-mon[21788]: Deploying daemon osd.2 on smithi083 2024-04-22T21:32:23.569 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:23 smithi083 ceph-mon[21788]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:23.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:23 smithi172 ceph-mon[23940]: Deploying daemon osd.2 on smithi083 2024-04-22T21:32:23.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:23 smithi172 ceph-mon[23940]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:24.837 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:24 smithi083 ceph-mon[21788]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:25.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:24 smithi172 ceph-mon[23940]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:26.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:26 smithi083 ceph-mon[21788]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:27.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:26 smithi172 ceph-mon[23940]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:28.950 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:28 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:28.950 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:28 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:28.950 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:28 smithi083 ceph-mon[21788]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:29.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:28 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:29.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:28 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:29.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:28 smithi172 ceph-mon[23940]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:30.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:30.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:32:30.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:30.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:32:30.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:30.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:32:30.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:30.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:32:31.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:31 smithi083 ceph-mon[21788]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:31.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:32:31.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:32:31.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:31 smithi172 ceph-mon[23940]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:31.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:32:31.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:32:32.577 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:32 smithi083 ceph-mon[21788]: from='osd.2 [v2:172.21.15.83:6818/2099285298,v1:172.21.15.83:6819/2099285298]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T21:32:32.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:32 smithi172 ceph-mon[23940]: from='osd.2 [v2:172.21.15.83:6818/2099285298,v1:172.21.15.83:6819/2099285298]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T21:32:32.607 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 2 on host 'smithi083' 2024-04-22T21:32:33.186 DEBUG:teuthology.orchestra.run.smithi083:osd.2> sudo journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T21:32:33.189 INFO:tasks.cephadm:Deploying osd.3 on smithi083 with /dev/vg_nvme/lv_1... 2024-04-22T21:32:33.189 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T21:32:33.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:33 smithi083 ceph-mon[21788]: from='osd.2 [v2:172.21.15.83:6818/2099285298,v1:172.21.15.83:6819/2099285298]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T21:32:33.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:33 smithi083 ceph-mon[21788]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T21:32:33.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:33 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:33.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:33 smithi083 ceph-mon[21788]: from='osd.2 [v2:172.21.15.83:6818/2099285298,v1:172.21.15.83:6819/2099285298]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-04-22T21:32:33.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:33 smithi083 ceph-mon[21788]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:33.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:33 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:33.478 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:32:33 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2[39878]: 2024-04-22T21:32:33.304+0000 7ff84173f700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T21:32:33.478 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:32:33 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2[39878]: 2024-04-22T21:32:33.307+0000 7ff83c0d6700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:32:33.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:33 smithi172 ceph-mon[23940]: from='osd.2 [v2:172.21.15.83:6818/2099285298,v1:172.21.15.83:6819/2099285298]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T21:32:33.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:33 smithi172 ceph-mon[23940]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T21:32:33.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:33 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:33.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:33 smithi172 ceph-mon[23940]: from='osd.2 [v2:172.21.15.83:6818/2099285298,v1:172.21.15.83:6819/2099285298]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-04-22T21:32:33.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:33 smithi172 ceph-mon[23940]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:33.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:33 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:34.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:34 smithi172 ceph-mon[23940]: from='osd.2 [v2:172.21.15.83:6818/2099285298,v1:172.21.15.83:6819/2099285298]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-04-22T21:32:34.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:34 smithi172 ceph-mon[23940]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T21:32:34.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:34 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:34.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:34 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:34.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:34 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:34.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:34 smithi083 ceph-mon[21788]: from='osd.2 [v2:172.21.15.83:6818/2099285298,v1:172.21.15.83:6819/2099285298]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-04-22T21:32:34.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:34 smithi083 ceph-mon[21788]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T21:32:34.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:34 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:34.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:34 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:34.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:34 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:35.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:35 smithi083 ceph-mon[21788]: purged_snaps scrub starts 2024-04-22T21:32:35.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:35 smithi083 ceph-mon[21788]: purged_snaps scrub ok 2024-04-22T21:32:35.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:35 smithi083 ceph-mon[21788]: osd.2 [v2:172.21.15.83:6818/2099285298,v1:172.21.15.83:6819/2099285298] boot 2024-04-22T21:32:35.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:35 smithi083 ceph-mon[21788]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T21:32:35.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:35 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:35.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:35 smithi083 ceph-mon[21788]: pgmap v69: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:35.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:35 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T21:32:35.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:35 smithi172 ceph-mon[23940]: purged_snaps scrub starts 2024-04-22T21:32:35.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:35 smithi172 ceph-mon[23940]: purged_snaps scrub ok 2024-04-22T21:32:35.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:35 smithi172 ceph-mon[23940]: osd.2 [v2:172.21.15.83:6818/2099285298,v1:172.21.15.83:6819/2099285298] boot 2024-04-22T21:32:35.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:35 smithi172 ceph-mon[23940]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T21:32:35.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:35 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:32:35.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:35 smithi172 ceph-mon[23940]: pgmap v69: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:32:35.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:35 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T21:32:36.194 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T21:32:36.194 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T21:32:36.194 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-04-22T21:32:36.194 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-04-22T21:32:36.194 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123958 s, 846 MB/s 2024-04-22T21:32:36.194 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-04-22T21:32:36.586 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:36 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T21:32:36.586 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:36 smithi083 ceph-mon[21788]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T21:32:36.586 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:36 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T21:32:36.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:36 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T21:32:36.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:36 smithi172 ceph-mon[23940]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T21:32:36.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:36 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T21:32:36.874 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_1 2024-04-22T21:32:37.398 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:37 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T21:32:37.398 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:37 smithi083 ceph-mon[21788]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T21:32:37.398 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:37 smithi083 ceph-mon[21788]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:37.398 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41172]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:32:37.398 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41172]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:32:37.398 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41172]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:32:37.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:37 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T21:32:37.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:37 smithi172 ceph-mon[23940]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T21:32:37.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:37 smithi172 ceph-mon[23940]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:37.650 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41215]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:32:37.650 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41215]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:32:37.650 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41215]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:32:37.935 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41223]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:32:37.936 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41223]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:32:37.936 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41223]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:32:38.186 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41299]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:32:38.186 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41274]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:32:38.186 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41274]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:32:38.187 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:32:37 smithi083 sudo[41274]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:32:38.459 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41299]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:32:38.459 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41299]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:32:38.460 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:38 smithi083 ceph-mon[21788]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T21:32:38.460 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:38.460 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:38.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:38 smithi172 ceph-mon[23940]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T21:32:38.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:38.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:38.714 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41353]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:32:38.715 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41353]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:32:38.715 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41353]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:32:38.974 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41400]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:32:38.975 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41400]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:32:38.975 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41400]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:32:39.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41412]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:32:39.271 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41412]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:32:39.272 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:38 smithi083 sudo[41412]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:32:39.475 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 sudo[26269]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:32:39.475 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 sudo[26269]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:32:39.475 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 sudo[26269]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:32:39.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:39 smithi083 ceph-mon[21788]: mgrmap e19: smithi083.ochghz(active, since 2m), standbys: smithi172.edmjti 2024-04-22T21:32:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:39 smithi083 ceph-mon[21788]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:32:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:39 smithi083 ceph-mon[21788]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:39 smithi083 ceph-mon[21788]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:32:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:39 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-04-22T21:32:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:39 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi172"}]: dispatch 2024-04-22T21:32:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:39 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-04-22T21:32:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:39 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi172"}]: dispatch 2024-04-22T21:32:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:39 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:32:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:39 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:32:39.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:39 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:39.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 sudo[26272]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:32:39.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 sudo[26272]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:32:39.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 sudo[26272]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:32:39.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 ceph-mon[23940]: mgrmap e19: smithi083.ochghz(active, since 2m), standbys: smithi172.edmjti 2024-04-22T21:32:39.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 ceph-mon[23940]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:32:39.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 ceph-mon[23940]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:39.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 ceph-mon[23940]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:32:39.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-04-22T21:32:39.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi172"}]: dispatch 2024-04-22T21:32:39.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-04-22T21:32:39.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mon metadata", "id": "smithi172"}]: dispatch 2024-04-22T21:32:39.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:32:39.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:32:39.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:39 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:40.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:40 smithi083 ceph-mon[21788]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:32:40.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:40 smithi083 ceph-mon[21788]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:32:40.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:40 smithi083 ceph-mon[21788]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:32:40.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:40 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:40.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:40 smithi172 ceph-mon[23940]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:32:40.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:40 smithi172 ceph-mon[23940]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:32:40.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:40 smithi172 ceph-mon[23940]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:32:40.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:40 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:41.628 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:41 smithi083 ceph-mon[21788]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:41.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:41 smithi172 ceph-mon[23940]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:43.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:43.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:32:43.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:32:43.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:32:43.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: Adjusting osd_memory_target on smithi083 to 4288M 2024-04-22T21:32:43.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:43.597 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:43.597 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:32:43.597 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:43.597 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:32:43.597 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:43.597 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:43.598 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/288235470' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "939a71f6-56fd-4b75-b0ec-58d7ba5deb3c"}]: dispatch 2024-04-22T21:32:43.598 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/288235470' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "939a71f6-56fd-4b75-b0ec-58d7ba5deb3c"}]': finished 2024-04-22T21:32:43.598 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T21:32:43.598 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:43 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:32:43.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: Adjusting osd_memory_target on smithi083 to 4288M 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/288235470' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "939a71f6-56fd-4b75-b0ec-58d7ba5deb3c"}]: dispatch 2024-04-22T21:32:43.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/288235470' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "939a71f6-56fd-4b75-b0ec-58d7ba5deb3c"}]': finished 2024-04-22T21:32:43.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T21:32:43.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:43 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:32:44.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:44 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/3246364126' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:32:44.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:44 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3246364126' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:32:45.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:45 smithi172 ceph-mon[23940]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:45.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:45 smithi083 ceph-mon[21788]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:46.652 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:46 smithi083 ceph-mon[21788]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:47.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:46 smithi172 ceph-mon[23940]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:49.367 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:49 smithi083 ceph-mon[21788]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:49.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:49 smithi172 ceph-mon[23940]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:50.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T21:32:50.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:50.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:50 smithi083 ceph-mon[21788]: Deploying daemon osd.3 on smithi083 2024-04-22T21:32:50.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T21:32:50.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:50.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:50 smithi172 ceph-mon[23940]: Deploying daemon osd.3 on smithi083 2024-04-22T21:32:51.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:51 smithi083 ceph-mon[21788]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:51 smithi172 ceph-mon[23940]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:52.851 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:52 smithi083 ceph-mon[21788]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:53.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:52 smithi172 ceph-mon[23940]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:55.798 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:55.799 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:32:55.799 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:55.799 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:32:55.799 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:55 smithi083 ceph-mon[21788]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:55.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:55.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:32:55.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:32:55.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:32:55.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:55 smithi172 ceph-mon[23940]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:56.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:56 smithi083 ceph-mon[21788]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:57.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:56 smithi172 ceph-mon[23940]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:57.256 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 3 on host 'smithi083' 2024-04-22T21:32:58.294 DEBUG:teuthology.orchestra.run.smithi083:osd.3> sudo journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T21:32:58.297 INFO:tasks.cephadm:Deploying osd.4 on smithi172 with /dev/vg_nvme/lv_4... 2024-04-22T21:32:58.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T21:32:58.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:58 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:58.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:58 smithi172 ceph-mon[23940]: from='osd.3 [v2:172.21.15.83:6826/678966071,v1:172.21.15.83:6827/678966071]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:32:58.644 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:58 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:32:58.644 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:58 smithi083 ceph-mon[21788]: from='osd.3 [v2:172.21.15.83:6826/678966071,v1:172.21.15.83:6827/678966071]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:32:59.530 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:59 smithi083 ceph-mon[21788]: from='osd.3 [v2:172.21.15.83:6826/678966071,v1:172.21.15.83:6827/678966071]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T21:32:59.530 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:59 smithi083 ceph-mon[21788]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T21:32:59.530 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:59 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:32:59.530 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:59 smithi083 ceph-mon[21788]: from='osd.3 [v2:172.21.15.83:6826/678966071,v1:172.21.15.83:6827/678966071]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-04-22T21:32:59.530 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:59 smithi083 ceph-mon[21788]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:59.530 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:59 smithi083 ceph-mon[21788]: from='osd.3 [v2:172.21.15.83:6826/678966071,v1:172.21.15.83:6827/678966071]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-04-22T21:32:59.531 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:59 smithi083 ceph-mon[21788]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T21:32:59.531 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:32:59 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:32:59.531 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:32:59 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3[43141]: 2024-04-22T21:32:59.280+0000 7fe61a212700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T21:32:59.531 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:32:59 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3[43141]: 2024-04-22T21:32:59.284+0000 7fe6143a8700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:32:59.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:59 smithi172 ceph-mon[23940]: from='osd.3 [v2:172.21.15.83:6826/678966071,v1:172.21.15.83:6827/678966071]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T21:32:59.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:59 smithi172 ceph-mon[23940]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T21:32:59.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:59 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:32:59.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:59 smithi172 ceph-mon[23940]: from='osd.3 [v2:172.21.15.83:6826/678966071,v1:172.21.15.83:6827/678966071]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-04-22T21:32:59.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:59 smithi172 ceph-mon[23940]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:32:59.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:59 smithi172 ceph-mon[23940]: from='osd.3 [v2:172.21.15.83:6826/678966071,v1:172.21.15.83:6827/678966071]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-04-22T21:32:59.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:59 smithi172 ceph-mon[23940]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T21:32:59.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:32:59 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:33:00.462 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T21:33:00.463 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T21:33:00.463 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-04-22T21:33:00.463 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-04-22T21:33:00.463 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112612 s, 931 MB/s 2024-04-22T21:33:00.463 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-04-22T21:33:00.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:00 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:33:00.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:00 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:33:00.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:00 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:33:00.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:00 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:33:01.051 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi172:vg_nvme/lv_4 2024-04-22T21:33:01.581 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:01 smithi083 ceph-mon[21788]: purged_snaps scrub starts 2024-04-22T21:33:01.581 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:01 smithi083 ceph-mon[21788]: purged_snaps scrub ok 2024-04-22T21:33:01.581 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:01 smithi083 ceph-mon[21788]: osd.3 [v2:172.21.15.83:6826/678966071,v1:172.21.15.83:6827/678966071] boot 2024-04-22T21:33:01.581 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:01 smithi083 ceph-mon[21788]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T21:33:01.581 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:01 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:33:01.581 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:01 smithi083 ceph-mon[21788]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:33:01.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:01 smithi172 ceph-mon[23940]: purged_snaps scrub starts 2024-04-22T21:33:01.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:01 smithi172 ceph-mon[23940]: purged_snaps scrub ok 2024-04-22T21:33:01.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:01 smithi172 ceph-mon[23940]: osd.3 [v2:172.21.15.83:6826/678966071,v1:172.21.15.83:6827/678966071] boot 2024-04-22T21:33:01.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:01 smithi172 ceph-mon[23940]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T21:33:01.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:01 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:33:01.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:01 smithi172 ceph-mon[23940]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:33:02.308 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:02 smithi172 ceph-mon[23940]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T21:33:02.308 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:02 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:02.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:02 smithi083 ceph-mon[21788]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T21:33:02.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:02 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:03.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:03 smithi083 ceph-mon[21788]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:03.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:03 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:33:03.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:03 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:33:03.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:03 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:03.548 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:03 smithi172 ceph-mon[23940]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:03.549 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:03 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:33:03.549 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:03 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:33:03.549 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:03 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:04.361 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:04 smithi172 ceph-mon[23940]: from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:33:04.495 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:04 smithi083 ceph-mon[21788]: from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:33:05.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: Detected new or changed devices on smithi083 2024-04-22T21:33:05.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:05.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:05.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:05.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:05.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: Adjusting osd_memory_target on smithi083 to 3216M 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='client.? 172.21.15.172:0/2401741432' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c67d9990-dd06-4746-a651-443fb865e423"}]: dispatch 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c67d9990-dd06-4746-a651-443fb865e423"}]: dispatch 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c67d9990-dd06-4746-a651-443fb865e423"}]': finished 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: osdmap e28: 5 total, 4 up, 5 in 2024-04-22T21:33:05.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:05 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:05.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: Detected new or changed devices on smithi083 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: Adjusting osd_memory_target on smithi083 to 3216M 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:05.901 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:05.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/2401741432' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c67d9990-dd06-4746-a651-443fb865e423"}]: dispatch 2024-04-22T21:33:05.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c67d9990-dd06-4746-a651-443fb865e423"}]: dispatch 2024-04-22T21:33:05.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c67d9990-dd06-4746-a651-443fb865e423"}]': finished 2024-04-22T21:33:05.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: osdmap e28: 5 total, 4 up, 5 in 2024-04-22T21:33:05.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:05 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:06.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:06 smithi172 ceph-mon[23940]: from='client.? 172.21.15.172:0/1406786321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:33:06.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:06 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/1406786321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:33:07.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:07 smithi172 ceph-mon[23940]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:07.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:07 smithi083 ceph-mon[21788]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:09.382 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:09 smithi172 ceph-mon[23940]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:09.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:09 smithi083 ceph-mon[21788]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:10.756 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:10 smithi172 ceph-mon[23940]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:10.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:10 smithi083 ceph-mon[21788]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:11.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T21:33:11.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:11.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:11 smithi083 ceph-mon[21788]: Deploying daemon osd.4 on smithi172 2024-04-22T21:33:11.938 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T21:33:11.938 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:11.938 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:11 smithi172 ceph-mon[23940]: Deploying daemon osd.4 on smithi172 2024-04-22T21:33:12.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:12 smithi083 ceph-mon[21788]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:12.935 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:12 smithi172 ceph-mon[23940]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:14.816 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:14 smithi172 ceph-mon[23940]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:14.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:14 smithi083 ceph-mon[21788]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:17.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:17.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:17.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:17.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:33:17.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:17 smithi083 ceph-mon[21788]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:17.769 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:17.770 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:17.770 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:17.770 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:33:17.770 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:17 smithi172 ceph-mon[23940]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:19.546 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:19 smithi172 ceph-mon[23940]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:19.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:19 smithi083 ceph-mon[21788]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:19.904 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 4 on host 'smithi172' 2024-04-22T21:33:20.438 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:20 smithi172 ceph-mon[23940]: from='osd.4 [v2:172.21.15.172:6800/2779155763,v1:172.21.15.172:6801/2779155763]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:33:20.438 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:20 smithi172 ceph-mon[23940]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:33:20.438 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:20 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:20.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:20 smithi083 ceph-mon[21788]: from='osd.4 [v2:172.21.15.172:6800/2779155763,v1:172.21.15.172:6801/2779155763]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:33:20.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:20 smithi083 ceph-mon[21788]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:33:20.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:20 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:20.861 DEBUG:teuthology.orchestra.run.smithi172:osd.4> sudo journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T21:33:20.863 INFO:tasks.cephadm:Deploying osd.5 on smithi172 with /dev/vg_nvme/lv_3... 2024-04-22T21:33:20.864 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T21:33:21.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:21 smithi172 ceph-mon[23940]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T21:33:21.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:21 smithi172 ceph-mon[23940]: osdmap e29: 5 total, 4 up, 5 in 2024-04-22T21:33:21.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:21.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:21 smithi172 ceph-mon[23940]: from='osd.4 [v2:172.21.15.172:6800/2779155763,v1:172.21.15.172:6801/2779155763]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:33:21.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:21 smithi172 ceph-mon[23940]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:33:21.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:21.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:21 smithi172 ceph-mon[23940]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:21.596 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:33:21 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:33:21.184+0000 7f69dcadf700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T21:33:21.596 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:33:21 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:33:21.189+0000 7f69d8c79700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:33:21.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:21 smithi083 ceph-mon[21788]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T21:33:21.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:21 smithi083 ceph-mon[21788]: osdmap e29: 5 total, 4 up, 5 in 2024-04-22T21:33:21.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:21.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:21 smithi083 ceph-mon[21788]: from='osd.4 [v2:172.21.15.172:6800/2779155763,v1:172.21.15.172:6801/2779155763]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:33:21.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:21 smithi083 ceph-mon[21788]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:33:21.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:21.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:21 smithi083 ceph-mon[21788]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:22.275 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:22 smithi172 ceph-mon[23940]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-04-22T21:33:22.275 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:22 smithi172 ceph-mon[23940]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T21:33:22.275 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:22 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:22.275 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:22 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:22.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:22 smithi083 ceph-mon[21788]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-04-22T21:33:22.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:22 smithi083 ceph-mon[21788]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T21:33:22.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:22 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:22.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:22 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:23.436 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:23 smithi172 ceph-mon[23940]: purged_snaps scrub starts 2024-04-22T21:33:23.436 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:23 smithi172 ceph-mon[23940]: purged_snaps scrub ok 2024-04-22T21:33:23.436 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:23 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:23.436 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:23 smithi172 ceph-mon[23940]: osd.4 [v2:172.21.15.172:6800/2779155763,v1:172.21.15.172:6801/2779155763] boot 2024-04-22T21:33:23.436 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:23 smithi172 ceph-mon[23940]: osdmap e31: 5 total, 5 up, 5 in 2024-04-22T21:33:23.436 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:23 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:23.436 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:23 smithi172 ceph-mon[23940]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:23.437 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:23 smithi172 ceph-mon[23940]: osdmap e32: 5 total, 5 up, 5 in 2024-04-22T21:33:23.476 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:23 smithi083 ceph-mon[21788]: purged_snaps scrub starts 2024-04-22T21:33:23.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:23 smithi083 ceph-mon[21788]: purged_snaps scrub ok 2024-04-22T21:33:23.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:23 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:23.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:23 smithi083 ceph-mon[21788]: osd.4 [v2:172.21.15.172:6800/2779155763,v1:172.21.15.172:6801/2779155763] boot 2024-04-22T21:33:23.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:23 smithi083 ceph-mon[21788]: osdmap e31: 5 total, 5 up, 5 in 2024-04-22T21:33:23.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:23 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:33:23.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:23 smithi083 ceph-mon[21788]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:33:23.477 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:23 smithi083 ceph-mon[21788]: osdmap e32: 5 total, 5 up, 5 in 2024-04-22T21:33:23.922 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T21:33:23.922 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T21:33:23.923 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-04-22T21:33:23.923 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-04-22T21:33:23.923 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124042 s, 845 MB/s 2024-04-22T21:33:23.923 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-04-22T21:33:24.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi172:vg_nvme/lv_3 2024-04-22T21:33:25.465 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:25 smithi172 ceph-mon[23940]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T21:33:25.465 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:25 smithi172 ceph-mon[23940]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:25.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:25 smithi083 ceph-mon[21788]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T21:33:25.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:25 smithi083 ceph-mon[21788]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:26.255 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:26 smithi172 ceph-mon[23940]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T21:33:26.255 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:26 smithi172 ceph-mon[23940]: Detected new or changed devices on smithi172 2024-04-22T21:33:26.255 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:26 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:26.255 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:26 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:26.255 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:26 smithi172 ceph-mon[23940]: Adjusting osd_memory_target on smithi172 to 15936M 2024-04-22T21:33:26.255 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:26 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:26.255 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:26 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:26.255 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:26 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:26.255 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:26 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:26.255 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:26 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:33:26.255 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:26 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:26.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:26 smithi083 ceph-mon[21788]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T21:33:26.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:26 smithi083 ceph-mon[21788]: Detected new or changed devices on smithi172 2024-04-22T21:33:26.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:26 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:26.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:26 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:26.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:26 smithi083 ceph-mon[21788]: Adjusting osd_memory_target on smithi172 to 15936M 2024-04-22T21:33:26.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:26 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:26.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:26 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:26.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:26 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:26.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:26 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:26.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:26 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:33:26.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:26 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:27.272 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:27 smithi172 ceph-mon[23940]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T21:33:27.272 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:27 smithi172 ceph-mon[23940]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T21:33:27.272 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:27 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:33:27.272 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:27 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:33:27.272 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:27 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:27.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:27 smithi083 ceph-mon[21788]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T21:33:27.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:27 smithi083 ceph-mon[21788]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T21:33:27.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:27 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:33:27.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:27 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:33:27.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:27 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:28.214 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:28 smithi172 ceph-mon[23940]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:33:28.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:28 smithi083 ceph-mon[21788]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:33:29.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:29 smithi172 ceph-mon[23940]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T21:33:29.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:29 smithi083 ceph-mon[21788]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T21:33:30.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:30 smithi172 ceph-mon[23940]: from='client.? 172.21.15.172:0/228369904' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "490a6258-1cf1-4c00-b819-c56c075b6c54"}]: dispatch 2024-04-22T21:33:30.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:30 smithi172 ceph-mon[23940]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "490a6258-1cf1-4c00-b819-c56c075b6c54"}]: dispatch 2024-04-22T21:33:30.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:30 smithi172 ceph-mon[23940]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "490a6258-1cf1-4c00-b819-c56c075b6c54"}]': finished 2024-04-22T21:33:30.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:30 smithi172 ceph-mon[23940]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T21:33:30.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:33:30.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:30 smithi172 ceph-mon[23940]: from='client.? 172.21.15.172:0/1809086359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:33:30.536 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:30 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/228369904' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "490a6258-1cf1-4c00-b819-c56c075b6c54"}]: dispatch 2024-04-22T21:33:30.536 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:30 smithi083 ceph-mon[21788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "490a6258-1cf1-4c00-b819-c56c075b6c54"}]: dispatch 2024-04-22T21:33:30.536 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:30 smithi083 ceph-mon[21788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "490a6258-1cf1-4c00-b819-c56c075b6c54"}]': finished 2024-04-22T21:33:30.536 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:30 smithi083 ceph-mon[21788]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T21:33:30.536 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:33:30.536 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:30 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/1809086359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:33:31.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:31 smithi172 ceph-mon[23940]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:33:31.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:33:31.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:33:31.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:31 smithi083 ceph-mon[21788]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:33:31.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:33:31.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:33:32.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:32 smithi172 ceph-mon[23940]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:32.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:32 smithi083 ceph-mon[21788]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:34.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:34 smithi172 ceph-mon[23940]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:34.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:34 smithi083 ceph-mon[21788]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:35.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:35 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T21:33:35.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:35 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:35.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:35 smithi172 ceph-mon[23940]: Deploying daemon osd.5 on smithi172 2024-04-22T21:33:35.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:35 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T21:33:35.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:35 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:35.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:35 smithi083 ceph-mon[21788]: Deploying daemon osd.5 on smithi172 2024-04-22T21:33:36.758 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:36 smithi172 ceph-mon[23940]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:36.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:36 smithi083 ceph-mon[21788]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:39.511 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:39 smithi172 ceph-mon[23940]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:39.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:39 smithi083 ceph-mon[21788]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:41.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:41 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:41.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:41 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:41.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:41 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:41.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:41 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:33:41.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:41 smithi083 ceph-mon[21788]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:41.738 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:41 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:41.738 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:41 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:41.738 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:41 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:41.738 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:41 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:33:41.739 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:41 smithi172 ceph-mon[23940]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:42.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:42 smithi172 ceph-mon[23940]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:42.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:42 smithi083 ceph-mon[21788]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:43.674 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 5 on host 'smithi172' 2024-04-22T21:33:44.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:44 smithi172 ceph-mon[23940]: from='osd.5 [v2:172.21.15.172:6808/542171958,v1:172.21.15.172:6809/542171958]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:33:44.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:44 smithi172 ceph-mon[23940]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:33:44.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:44 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:44.380 DEBUG:teuthology.orchestra.run.smithi172:osd.5> sudo journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T21:33:44.382 INFO:tasks.cephadm:Deploying osd.6 on smithi172 with /dev/vg_nvme/lv_2... 2024-04-22T21:33:44.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T21:33:44.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:44 smithi083 ceph-mon[21788]: from='osd.5 [v2:172.21.15.172:6808/542171958,v1:172.21.15.172:6809/542171958]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:33:44.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:44 smithi083 ceph-mon[21788]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:33:44.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:44 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:45.428 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:33:45 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:33:45.190+0000 7f57eae0c700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T21:33:45.428 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:33:45 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:33:45.196+0000 7f57e6fa6700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:33:45.428 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:45 smithi172 ceph-mon[23940]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T21:33:45.428 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:45 smithi172 ceph-mon[23940]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T21:33:45.428 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:45 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:33:45.428 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:45 smithi172 ceph-mon[23940]: from='osd.5 [v2:172.21.15.172:6808/542171958,v1:172.21.15.172:6809/542171958]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:33:45.428 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:45 smithi172 ceph-mon[23940]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:33:45.428 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:45 smithi172 ceph-mon[23940]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:45.574 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:45 smithi083 ceph-mon[21788]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T21:33:45.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:45 smithi083 ceph-mon[21788]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T21:33:45.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:45 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:33:45.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:45 smithi083 ceph-mon[21788]: from='osd.5 [v2:172.21.15.172:6808/542171958,v1:172.21.15.172:6809/542171958]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:33:45.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:45 smithi083 ceph-mon[21788]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:33:45.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:45 smithi083 ceph-mon[21788]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:33:46.267 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:46 smithi172 ceph-mon[23940]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-04-22T21:33:46.267 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:46 smithi172 ceph-mon[23940]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T21:33:46.267 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:46 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:33:46.267 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:46 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:33:46.267 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:46 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:46.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:46 smithi083 ceph-mon[21788]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-04-22T21:33:46.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:46 smithi083 ceph-mon[21788]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T21:33:46.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:46 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:33:46.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:46 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:33:46.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:46 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:46.962 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T21:33:46.962 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T21:33:46.962 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-04-22T21:33:46.962 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-04-22T21:33:46.962 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130793 s, 802 MB/s 2024-04-22T21:33:46.962 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-04-22T21:33:47.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:47 smithi172 ceph-mon[23940]: purged_snaps scrub starts 2024-04-22T21:33:47.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:47 smithi172 ceph-mon[23940]: purged_snaps scrub ok 2024-04-22T21:33:47.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:47 smithi172 ceph-mon[23940]: osd.5 [v2:172.21.15.172:6808/542171958,v1:172.21.15.172:6809/542171958] boot 2024-04-22T21:33:47.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:47 smithi172 ceph-mon[23940]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T21:33:47.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:47 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:33:47.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:47 smithi172 ceph-mon[23940]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:33:47.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:47 smithi083 ceph-mon[21788]: purged_snaps scrub starts 2024-04-22T21:33:47.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:47 smithi083 ceph-mon[21788]: purged_snaps scrub ok 2024-04-22T21:33:47.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:47 smithi083 ceph-mon[21788]: osd.5 [v2:172.21.15.172:6808/542171958,v1:172.21.15.172:6809/542171958] boot 2024-04-22T21:33:47.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:47 smithi083 ceph-mon[21788]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T21:33:47.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:47 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:33:47.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:47 smithi083 ceph-mon[21788]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:33:48.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi172:vg_nvme/lv_2 2024-04-22T21:33:48.291 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:48 smithi172 ceph-mon[23940]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T21:33:48.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:48 smithi083 ceph-mon[21788]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T21:33:49.435 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:49 smithi172 ceph-mon[23940]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T21:33:49.435 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:49 smithi172 ceph-mon[23940]: pgmap v128: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T21:33:49.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:49 smithi083 ceph-mon[21788]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T21:33:49.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:49 smithi083 ceph-mon[21788]: pgmap v128: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T21:33:50.470 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:50 smithi172 ceph-mon[23940]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-22T21:33:50.471 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:50 smithi172 ceph-mon[23940]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T21:33:50.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:50 smithi083 ceph-mon[21788]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-22T21:33:50.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:50 smithi083 ceph-mon[21788]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T21:33:51.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: Detected new or changed devices on smithi172 2024-04-22T21:33:51.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T21:33:51.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:51.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:51.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:51.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: Adjusting osd_memory_target on smithi172 to 7968M 2024-04-22T21:33:51.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:51.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:51.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:51.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:51.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:33:51.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:51.346 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:51 smithi172 ceph-mon[23940]: pgmap v131: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T21:33:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: Detected new or changed devices on smithi172 2024-04-22T21:33:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T21:33:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:33:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: Adjusting osd_memory_target on smithi172 to 7968M 2024-04-22T21:33:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:33:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:33:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:51 smithi083 ceph-mon[21788]: pgmap v131: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T21:33:52.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:52 smithi172 ceph-mon[23940]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:33:52.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:52 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:33:52.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:52 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:33:52.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:52 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:52.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:52 smithi083 ceph-mon[21788]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:33:52.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:52 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:33:52.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:52 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:33:52.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:52 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:53.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:53 smithi172 ceph-mon[23940]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:33:53.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:53 smithi083 ceph-mon[21788]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:33:54.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:54 smithi172 ceph-mon[23940]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-22T21:33:54.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:54 smithi172 ceph-mon[23940]: Cluster is now healthy 2024-04-22T21:33:54.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:54 smithi172 ceph-mon[23940]: from='client.? 172.21.15.172:0/1946286386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "351cdd76-9bac-464a-84ff-ed1841688ea0"}]: dispatch 2024-04-22T21:33:54.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:54 smithi172 ceph-mon[23940]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "351cdd76-9bac-464a-84ff-ed1841688ea0"}]: dispatch 2024-04-22T21:33:54.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:54 smithi172 ceph-mon[23940]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "351cdd76-9bac-464a-84ff-ed1841688ea0"}]': finished 2024-04-22T21:33:54.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:54 smithi172 ceph-mon[23940]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T21:33:54.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:54 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:33:54.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:54 smithi172 ceph-mon[23940]: from='client.? 172.21.15.172:0/3741066553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:33:54.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:54 smithi083 ceph-mon[21788]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-22T21:33:54.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:54 smithi083 ceph-mon[21788]: Cluster is now healthy 2024-04-22T21:33:54.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:54 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/1946286386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "351cdd76-9bac-464a-84ff-ed1841688ea0"}]: dispatch 2024-04-22T21:33:54.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:54 smithi083 ceph-mon[21788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "351cdd76-9bac-464a-84ff-ed1841688ea0"}]: dispatch 2024-04-22T21:33:54.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:54 smithi083 ceph-mon[21788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "351cdd76-9bac-464a-84ff-ed1841688ea0"}]': finished 2024-04-22T21:33:54.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:54 smithi083 ceph-mon[21788]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T21:33:54.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:54 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:33:54.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:54 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/3741066553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:33:55.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:55 smithi172 ceph-mon[23940]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:33:55.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:55 smithi083 ceph-mon[21788]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:33:56.782 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:56 smithi172 ceph-mon[23940]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T21:33:56.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:56 smithi083 ceph-mon[21788]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T21:33:59.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:33:59 smithi172 ceph-mon[23940]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-22T21:33:59.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:33:59 smithi083 ceph-mon[21788]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-22T21:34:00.453 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:00 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T21:34:00.453 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:00 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:00.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:00 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T21:34:00.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:00 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:01.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:01 smithi172 ceph-mon[23940]: Deploying daemon osd.6 on smithi172 2024-04-22T21:34:01.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:01 smithi172 ceph-mon[23940]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T21:34:01.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:01 smithi083 ceph-mon[21788]: Deploying daemon osd.6 on smithi172 2024-04-22T21:34:01.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:01 smithi083 ceph-mon[21788]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T21:34:02.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:02 smithi172 ceph-mon[23940]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:34:02.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:02 smithi083 ceph-mon[21788]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:34:04.736 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:04 smithi172 ceph-mon[23940]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:34:04.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:04 smithi083 ceph-mon[21788]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:34:06.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:06 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:06.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:06 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:06.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:06 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:06.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:06 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:06.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:06 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:06.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:06 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:06.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:06 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:06.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:06 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:07.312 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:07 smithi172 ceph-mon[23940]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:34:07.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:07 smithi083 ceph-mon[21788]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:34:08.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:08 smithi172 ceph-mon[23940]: from='osd.6 [v2:172.21.15.172:6816/2944243465,v1:172.21.15.172:6817/2944243465]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T21:34:08.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:08 smithi172 ceph-mon[23940]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T21:34:08.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:08 smithi083 ceph-mon[21788]: from='osd.6 [v2:172.21.15.172:6816/2944243465,v1:172.21.15.172:6817/2944243465]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T21:34:08.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:08 smithi083 ceph-mon[21788]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T21:34:08.691 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 6 on host 'smithi172' 2024-04-22T21:34:09.332 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:09 smithi172 ceph-mon[23940]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T21:34:09.332 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:09 smithi172 ceph-mon[23940]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T21:34:09.332 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:09 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:34:09.332 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:09 smithi172 ceph-mon[23940]: from='osd.6 [v2:172.21.15.172:6816/2944243465,v1:172.21.15.172:6817/2944243465]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:34:09.332 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:09 smithi172 ceph-mon[23940]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:34:09.333 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:09 smithi172 ceph-mon[23940]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:34:09.333 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:09 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:09.355 DEBUG:teuthology.orchestra.run.smithi172:osd.6> sudo journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.6.service 2024-04-22T21:34:09.358 INFO:tasks.cephadm:Deploying osd.7 on smithi172 with /dev/vg_nvme/lv_1... 2024-04-22T21:34:09.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T21:34:09.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:09 smithi083 ceph-mon[21788]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T21:34:09.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:09 smithi083 ceph-mon[21788]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T21:34:09.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:09 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:34:09.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:09 smithi083 ceph-mon[21788]: from='osd.6 [v2:172.21.15.172:6816/2944243465,v1:172.21.15.172:6817/2944243465]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:34:09.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:09 smithi083 ceph-mon[21788]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:34:09.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:09 smithi083 ceph-mon[21788]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:34:09.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:09 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:10.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:10 smithi172 ceph-mon[23940]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-04-22T21:34:10.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:10 smithi172 ceph-mon[23940]: osdmap e46: 7 total, 6 up, 7 in 2024-04-22T21:34:10.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:10 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:34:10.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:10 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:34:10.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:10 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:10.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:10 smithi083 ceph-mon[21788]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-04-22T21:34:10.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:10 smithi083 ceph-mon[21788]: osdmap e46: 7 total, 6 up, 7 in 2024-04-22T21:34:10.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:10 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:34:10.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:10 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:34:10.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:10 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:11.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:11 smithi172 ceph-mon[23940]: purged_snaps scrub starts 2024-04-22T21:34:11.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:11 smithi172 ceph-mon[23940]: purged_snaps scrub ok 2024-04-22T21:34:11.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:11 smithi172 ceph-mon[23940]: osd.6 [v2:172.21.15.172:6816/2944243465,v1:172.21.15.172:6817/2944243465] boot 2024-04-22T21:34:11.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:11 smithi172 ceph-mon[23940]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T21:34:11.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:34:11.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:11 smithi172 ceph-mon[23940]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:11.525 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T21:34:11.525 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T21:34:11.525 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-04-22T21:34:11.525 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-04-22T21:34:11.525 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140386 s, 747 MB/s 2024-04-22T21:34:11.525 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-04-22T21:34:11.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:11 smithi083 ceph-mon[21788]: purged_snaps scrub starts 2024-04-22T21:34:11.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:11 smithi083 ceph-mon[21788]: purged_snaps scrub ok 2024-04-22T21:34:11.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:11 smithi083 ceph-mon[21788]: osd.6 [v2:172.21.15.172:6816/2944243465,v1:172.21.15.172:6817/2944243465] boot 2024-04-22T21:34:11.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:11 smithi083 ceph-mon[21788]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T21:34:11.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:34:11.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:11 smithi083 ceph-mon[21788]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:12.548 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:12 smithi172 ceph-mon[23940]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T21:34:12.576 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi172:vg_nvme/lv_1 2024-04-22T21:34:12.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:12 smithi083 ceph-mon[21788]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T21:34:13.432 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:13 smithi172 ceph-mon[23940]: osdmap e49: 7 total, 7 up, 7 in 2024-04-22T21:34:13.433 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:13 smithi172 ceph-mon[23940]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:13.433 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:13 smithi172 ceph-mon[23940]: osdmap e50: 7 total, 7 up, 7 in 2024-04-22T21:34:13.476 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:13 smithi083 ceph-mon[21788]: osdmap e49: 7 total, 7 up, 7 in 2024-04-22T21:34:13.476 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:13 smithi083 ceph-mon[21788]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:13.476 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:13 smithi083 ceph-mon[21788]: osdmap e50: 7 total, 7 up, 7 in 2024-04-22T21:34:15.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:15 smithi172 ceph-mon[23940]: osdmap e51: 7 total, 7 up, 7 in 2024-04-22T21:34:15.345 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:15 smithi172 ceph-mon[23940]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:15.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:15 smithi083 ceph-mon[21788]: osdmap e51: 7 total, 7 up, 7 in 2024-04-22T21:34:15.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:15 smithi083 ceph-mon[21788]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:16.245 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:34:16.245 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:34:16.245 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:16.245 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:16.245 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:16.245 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:16.245 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:16.245 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:16.245 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:16.245 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:16.245 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:16.246 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:16.246 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:16 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:16.552 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:34:16.552 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:34:16.552 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:16.552 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:16.552 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:16.552 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:16.552 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:16.552 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:16.552 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:16.552 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:16.552 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:16.553 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:16.553 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:16 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:17.385 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:17 smithi172 ceph-mon[23940]: from='client.24195 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:17.385 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:17 smithi172 ceph-mon[23940]: Detected new or changed devices on smithi172 2024-04-22T21:34:17.385 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:17 smithi172 ceph-mon[23940]: Adjusting osd_memory_target on smithi172 to 5312M 2024-04-22T21:34:17.385 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:17 smithi172 ceph-mon[23940]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T21:34:17.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:17 smithi083 ceph-mon[21788]: from='client.24195 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:17.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:17 smithi083 ceph-mon[21788]: Detected new or changed devices on smithi172 2024-04-22T21:34:17.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:17 smithi083 ceph-mon[21788]: Adjusting osd_memory_target on smithi172 to 5312M 2024-04-22T21:34:17.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:17 smithi083 ceph-mon[21788]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T21:34:18.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:18 smithi172 ceph-mon[23940]: from='client.? 172.21.15.172:0/3905349659' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53d58400-5ee6-4fdb-87a3-6f10b10c8c7e"}]: dispatch 2024-04-22T21:34:18.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:18 smithi172 ceph-mon[23940]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53d58400-5ee6-4fdb-87a3-6f10b10c8c7e"}]: dispatch 2024-04-22T21:34:18.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:18 smithi172 ceph-mon[23940]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53d58400-5ee6-4fdb-87a3-6f10b10c8c7e"}]': finished 2024-04-22T21:34:18.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:18 smithi172 ceph-mon[23940]: osdmap e52: 8 total, 7 up, 8 in 2024-04-22T21:34:18.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:18 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:34:18.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:18 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/3905349659' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53d58400-5ee6-4fdb-87a3-6f10b10c8c7e"}]: dispatch 2024-04-22T21:34:18.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:18 smithi083 ceph-mon[21788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53d58400-5ee6-4fdb-87a3-6f10b10c8c7e"}]: dispatch 2024-04-22T21:34:18.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:18 smithi083 ceph-mon[21788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53d58400-5ee6-4fdb-87a3-6f10b10c8c7e"}]': finished 2024-04-22T21:34:18.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:18 smithi083 ceph-mon[21788]: osdmap e52: 8 total, 7 up, 8 in 2024-04-22T21:34:18.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:18 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:34:19.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:19 smithi172 ceph-mon[23940]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:34:19.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:19 smithi172 ceph-mon[23940]: from='client.? 172.21.15.172:0/1124345407' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:34:19.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:19 smithi083 ceph-mon[21788]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:34:19.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:19 smithi083 ceph-mon[21788]: from='client.? 172.21.15.172:0/1124345407' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:34:20.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:20 smithi172 ceph-mon[23940]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T21:34:20.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:20 smithi083 ceph-mon[21788]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T21:34:22.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:22 smithi083 ceph-mon[21788]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T21:34:22.922 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:22 smithi172 ceph-mon[23940]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T21:34:24.727 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:24 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T21:34:24.727 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:24 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:24.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:24 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T21:34:24.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:24 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:25.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:25 smithi172 ceph-mon[23940]: Deploying daemon osd.7 on smithi172 2024-04-22T21:34:25.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:25 smithi172 ceph-mon[23940]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T21:34:25.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:25 smithi083 ceph-mon[21788]: Deploying daemon osd.7 on smithi172 2024-04-22T21:34:25.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:25 smithi083 ceph-mon[21788]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T21:34:26.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:26 smithi172 ceph-mon[23940]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:26.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:26 smithi083 ceph-mon[21788]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:29.417 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:29 smithi172 ceph-mon[23940]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:29.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:29 smithi083 ceph-mon[21788]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:31.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:31.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:31.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:31.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:31.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:30 smithi172 ceph-mon[23940]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:31.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:34:31.096 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:34:31.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:31.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:31.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:31.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:31.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:30 smithi083 ceph-mon[21788]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:31.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:34:31.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:34:32.808 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:32 smithi172 ceph-mon[23940]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:32.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:32 smithi083 ceph-mon[21788]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:33.571 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 7 on host 'smithi172' 2024-04-22T21:34:33.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:33 smithi172 ceph-mon[23940]: from='osd.7 [v2:172.21.15.172:6824/3721092024,v1:172.21.15.172:6825/3721092024]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T21:34:33.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:33 smithi172 ceph-mon[23940]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T21:34:33.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:33 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:33.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:33 smithi083 ceph-mon[21788]: from='osd.7 [v2:172.21.15.172:6824/3721092024,v1:172.21.15.172:6825/3721092024]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T21:34:33.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:33 smithi083 ceph-mon[21788]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T21:34:33.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:33 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:34.209 DEBUG:teuthology.orchestra.run.smithi172:osd.7> sudo journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.7.service 2024-04-22T21:34:34.211 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-22T21:34:34.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T21:34:34.778 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:34 smithi083 ceph-mon[21788]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T21:34:34.778 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:34 smithi083 ceph-mon[21788]: osdmap e53: 8 total, 7 up, 8 in 2024-04-22T21:34:34.779 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:34 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:34:34.779 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:34 smithi083 ceph-mon[21788]: from='osd.7 [v2:172.21.15.172:6824/3721092024,v1:172.21.15.172:6825/3721092024]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:34:34.779 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:34 smithi083 ceph-mon[21788]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:34:34.779 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:34 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:34.779 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:34 smithi083 ceph-mon[21788]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:34.962 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:34 smithi172 ceph-mon[23940]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T21:34:34.962 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:34 smithi172 ceph-mon[23940]: osdmap e53: 8 total, 7 up, 8 in 2024-04-22T21:34:34.962 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:34 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:34:34.962 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:34 smithi172 ceph-mon[23940]: from='osd.7 [v2:172.21.15.172:6824/3721092024,v1:172.21.15.172:6825/3721092024]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:34:34.962 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:34 smithi172 ceph-mon[23940]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-04-22T21:34:34.962 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:34 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:34.962 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:34 smithi172 ceph-mon[23940]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:34.962 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:34:34 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:34:34.666+0000 7f8ee8fc6700 -1 osd.7 0 waiting for initial osdmap 2024-04-22T21:34:34.963 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:34:34 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:34:34.673+0000 7f8ee495f700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:34:35.853 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:35 smithi083 ceph-mon[21788]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-04-22T21:34:35.853 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:35 smithi083 ceph-mon[21788]: osdmap e54: 8 total, 7 up, 8 in 2024-04-22T21:34:35.853 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:35 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:34:35.853 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:35 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:34:35.906 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:35 smithi172 ceph-mon[23940]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-04-22T21:34:35.906 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:35 smithi172 ceph-mon[23940]: osdmap e54: 8 total, 7 up, 8 in 2024-04-22T21:34:35.906 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:35 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:34:35.907 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:35 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:34:36.221 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:34:36.747 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:36 smithi083 ceph-mon[21788]: purged_snaps scrub starts 2024-04-22T21:34:36.748 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:36 smithi083 ceph-mon[21788]: purged_snaps scrub ok 2024-04-22T21:34:36.748 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:36 smithi083 ceph-mon[21788]: osd.7 [v2:172.21.15.172:6824/3721092024,v1:172.21.15.172:6825/3721092024] boot 2024-04-22T21:34:36.748 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:36 smithi083 ceph-mon[21788]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T21:34:36.748 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:36 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:34:36.748 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:36 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3483397737' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T21:34:36.748 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:36 smithi083 ceph-mon[21788]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:36.748 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:36 smithi083 ceph-mon[21788]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T21:34:36.873 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1713821675,"num_in_osds":8,"osd_in_since":1713821658,"num_remapped_pgs":1} 2024-04-22T21:34:36.874 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T21:34:37.094 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:37.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:36 smithi172 ceph-mon[23940]: purged_snaps scrub starts 2024-04-22T21:34:37.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:36 smithi172 ceph-mon[23940]: purged_snaps scrub ok 2024-04-22T21:34:37.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:36 smithi172 ceph-mon[23940]: osd.7 [v2:172.21.15.172:6824/3721092024,v1:172.21.15.172:6825/3721092024] boot 2024-04-22T21:34:37.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:36 smithi172 ceph-mon[23940]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T21:34:37.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:36 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:34:37.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:36 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/3483397737' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T21:34:37.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:36 smithi172 ceph-mon[23940]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:34:37.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:36 smithi172 ceph-mon[23940]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T21:34:38.767 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:34:38.767 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":58,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","created":"2024-04-22T21:28:48.803761+0000","modified":"2024-04-22T21:34:38.182582+0000","last_up_change":"2024-04-22T21:34:35.663841+0000","last_in_change":"2024-04-22T21:34:18.124697+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T21:32:34.853617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"120f52f4-cd77-4920-9b7f-9e02e652e3e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6803","nonce":2774972645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6805","nonce":2774972645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6809","nonce":2774972645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6807","nonce":2774972645}]},"public_addr":"172.21.15.83:6803/2774972645","cluster_addr":"172.21.15.83:6805/2774972645","heartbeat_back_addr":"172.21.15.83:6809/2774972645","heartbeat_front_addr":"172.21.15.83:6807/2774972645","state":["exists","up"]},{"osd":1,"uuid":"e885f222-674f-4a38-83bb-ab008b20607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6811","nonce":2475641293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6813","nonce":2475641293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6817","nonce":2475641293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6815","nonce":2475641293}]},"public_addr":"172.21.15.83:6811/2475641293","cluster_addr":"172.21.15.83:6813/2475641293","heartbeat_back_addr":"172.21.15.83:6817/2475641293","heartbeat_front_addr":"172.21.15.83:6815/2475641293","state":["exists","up"]},{"osd":2,"uuid":"ad91da27-115e-4ffc-8885-a6eb7fd412bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6819","nonce":2099285298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6821","nonce":2099285298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6825","nonce":2099285298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6823","nonce":2099285298}]},"public_addr":"172.21.15.83:6819/2099285298","cluster_addr":"172.21.15.83:6821/2099285298","heartbeat_back_addr":"172.21.15.83:6825/2099285298","heartbeat_front_addr":"172.21.15.83:6823/2099285298","state":["exists","up"]},{"osd":3,"uuid":"939a71f6-56fd-4b75-b0ec-58d7ba5deb3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6827","nonce":678966071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6829","nonce":678966071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6833","nonce":678966071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6831","nonce":678966071}]},"public_addr":"172.21.15.83:6827/678966071","cluster_addr":"172.21.15.83:6829/678966071","heartbeat_back_addr":"172.21.15.83:6833/678966071","heartbeat_front_addr":"172.21.15.83:6831/678966071","state":["exists","up"]},{"osd":4,"uuid":"c67d9990-dd06-4746-a651-443fb865e423","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6801","nonce":2779155763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6803","nonce":2779155763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6807","nonce":2779155763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6805","nonce":2779155763}]},"public_addr":"172.21.15.172:6801/2779155763","cluster_addr":"172.21.15.172:6803/2779155763","heartbeat_back_addr":"172.21.15.172:6807/2779155763","heartbeat_front_addr":"172.21.15.172:6805/2779155763","state":["exists","up"]},{"osd":5,"uuid":"490a6258-1cf1-4c00-b819-c56c075b6c54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6809","nonce":542171958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6811","nonce":542171958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6815","nonce":542171958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6813","nonce":542171958}]},"public_addr":"172.21.15.172:6809/542171958","cluster_addr":"172.21.15.172:6811/542171958","heartbeat_back_addr":"172.21.15.172:6815/542171958","heartbeat_front_addr":"172.21.15.172:6813/542171958","state":["exists","up"]},{"osd":6,"uuid":"351cdd76-9bac-464a-84ff-ed1841688ea0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6817","nonce":2944243465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6819","nonce":2944243465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6823","nonce":2944243465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6821","nonce":2944243465}]},"public_addr":"172.21.15.172:6817/2944243465","cluster_addr":"172.21.15.172:6819/2944243465","heartbeat_back_addr":"172.21.15.172:6823/2944243465","heartbeat_front_addr":"172.21.15.172:6821/2944243465","state":["exists","up"]},{"osd":7,"uuid":"53d58400-5ee6-4fdb-87a3-6f10b10c8c7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6825","nonce":3721092024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6827","nonce":3721092024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6831","nonce":3721092024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6829","nonce":3721092024}]},"public_addr":"172.21.15.172:6825/3721092024","cluster_addr":"172.21.15.172:6827/3721092024","heartbeat_back_addr":"172.21.15.172:6831/3721092024","heartbeat_front_addr":"172.21.15.172:6829/3721092024","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:31:36.162215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:32:06.946929+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:32:33.223740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:32:58.600373+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:33:20.542112+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:33:44.501084+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:34:09.141620+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:34:33.864057+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:0/987860463":"2024-04-23T21:30:29.546191+0000","172.21.15.83:0/1419744190":"2024-04-23T21:30:29.546191+0000","172.21.15.83:6800/1063515625":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/4011448407":"2024-04-23T21:30:29.546191+0000","172.21.15.83:0/3651180963":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/3206574196":"2024-04-23T21:29:41.896421+0000","172.21.15.83:6801/121189037":"2024-04-23T21:30:29.546191+0000","172.21.15.83:6801/1063515625":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/414614811":"2024-04-23T21:29:12.027949+0000","172.21.15.83:6800/121189037":"2024-04-23T21:30:29.546191+0000","172.21.15.83:6801/29670807":"2024-04-23T21:29:12.027949+0000","172.21.15.83:0/3198040173":"2024-04-23T21:29:12.027949+0000","172.21.15.83:0/1929740090":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/4155452649":"2024-04-23T21:29:12.027949+0000","172.21.15.83:6800/29670807":"2024-04-23T21:29:12.027949+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: Detected new or changed devices on smithi172 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: Adjusting osd_memory_target on smithi172 to 3984M 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T21:34:38.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:38 smithi083 ceph-mon[21788]: osdmap e58: 8 total, 8 up, 8 in 2024-04-22T21:34:38.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: Detected new or changed devices on smithi172 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: Adjusting osd_memory_target on smithi172 to 3984M 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:34:38.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T21:34:38.846 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:38 smithi172 ceph-mon[23940]: osdmap e58: 8 total, 8 up, 8 in 2024-04-22T21:34:39.407 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T21:32:34.853617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-22T21:34:39.407 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T21:34:39.624 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:39.649 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:39 smithi083 ceph-mon[21788]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:34:39.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:39 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/4258743834' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:34:39.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:39 smithi083 ceph-mon[21788]: osdmap e59: 8 total, 8 up, 8 in 2024-04-22T21:34:39.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:39 smithi172 ceph-mon[23940]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:34:39.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:39 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/4258743834' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:34:39.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:39 smithi172 ceph-mon[23940]: osdmap e59: 8 total, 8 up, 8 in 2024-04-22T21:34:40.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:40 smithi083 ceph-mon[21788]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:34:40.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:40 smithi172 ceph-mon[23940]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:34:41.325 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-04-22T21:34:41.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:41 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/2400999220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T21:34:41.929 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T21:34:41.929 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T21:34:42.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:41 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/2400999220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T21:34:42.853 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:42 smithi083 ceph-mon[21788]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-22T21:34:43.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:42 smithi172 ceph-mon[23940]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-22T21:34:43.873 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-04-22T21:34:43.873 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQDz1yZmvj3wNBAAgxSF+cTRcwgCWBjx9s6KMQ== 2024-04-22T21:34:44.515 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:44 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3751366937' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:34:44.515 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:44 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3751366937' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:34:44.539 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-04-22T21:34:44.540 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T21:34:44.540 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T21:34:44.577 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T21:34:44.577 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T21:34:44.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T21:34:44.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:44 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/3751366937' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:34:44.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:44 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/3751366937' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:34:44.826 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:45.208 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:45 smithi083 ceph-mon[21788]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-22T21:34:45.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:45 smithi172 ceph-mon[23940]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-22T21:34:46.688 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:34:46.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:46 smithi083 ceph-mon[21788]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T21:34:47.009 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:46 smithi172 ceph-mon[23940]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T21:34:47.350 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"active_gid":14198,"active_name":"smithi083.ochghz","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":3501584244},{"type":"v1","addr":"172.21.15.83:6801","nonce":3501584244}]},"active_addr":"172.21.15.83:6801/3501584244","active_change":"2024-04-22T21:30:29.546527+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14218,"name":"smithi172.edmjti","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.83:8443/","prometheus":"http://172.21.15.83:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":1117982058}]},{"addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3257626040}]},{"addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":2414828856}]},{"addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":4280868831}]}]}} 2024-04-22T21:34:47.356 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T21:34:47.356 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T21:34:47.356 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T21:34:47.576 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:47.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:47 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/80815080' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T21:34:47.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:47 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/80815080' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T21:34:49.373 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:34:49.374 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":59,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","created":"2024-04-22T21:28:48.803761+0000","modified":"2024-04-22T21:34:39.189348+0000","last_up_change":"2024-04-22T21:34:35.663841+0000","last_in_change":"2024-04-22T21:34:18.124697+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T21:32:34.853617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"120f52f4-cd77-4920-9b7f-9e02e652e3e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6803","nonce":2774972645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6805","nonce":2774972645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6809","nonce":2774972645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6807","nonce":2774972645}]},"public_addr":"172.21.15.83:6803/2774972645","cluster_addr":"172.21.15.83:6805/2774972645","heartbeat_back_addr":"172.21.15.83:6809/2774972645","heartbeat_front_addr":"172.21.15.83:6807/2774972645","state":["exists","up"]},{"osd":1,"uuid":"e885f222-674f-4a38-83bb-ab008b20607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6811","nonce":2475641293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6813","nonce":2475641293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6817","nonce":2475641293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6815","nonce":2475641293}]},"public_addr":"172.21.15.83:6811/2475641293","cluster_addr":"172.21.15.83:6813/2475641293","heartbeat_back_addr":"172.21.15.83:6817/2475641293","heartbeat_front_addr":"172.21.15.83:6815/2475641293","state":["exists","up"]},{"osd":2,"uuid":"ad91da27-115e-4ffc-8885-a6eb7fd412bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6819","nonce":2099285298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6821","nonce":2099285298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6825","nonce":2099285298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6823","nonce":2099285298}]},"public_addr":"172.21.15.83:6819/2099285298","cluster_addr":"172.21.15.83:6821/2099285298","heartbeat_back_addr":"172.21.15.83:6825/2099285298","heartbeat_front_addr":"172.21.15.83:6823/2099285298","state":["exists","up"]},{"osd":3,"uuid":"939a71f6-56fd-4b75-b0ec-58d7ba5deb3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6827","nonce":678966071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6829","nonce":678966071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6833","nonce":678966071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6831","nonce":678966071}]},"public_addr":"172.21.15.83:6827/678966071","cluster_addr":"172.21.15.83:6829/678966071","heartbeat_back_addr":"172.21.15.83:6833/678966071","heartbeat_front_addr":"172.21.15.83:6831/678966071","state":["exists","up"]},{"osd":4,"uuid":"c67d9990-dd06-4746-a651-443fb865e423","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6801","nonce":2779155763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6803","nonce":2779155763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6807","nonce":2779155763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6805","nonce":2779155763}]},"public_addr":"172.21.15.172:6801/2779155763","cluster_addr":"172.21.15.172:6803/2779155763","heartbeat_back_addr":"172.21.15.172:6807/2779155763","heartbeat_front_addr":"172.21.15.172:6805/2779155763","state":["exists","up"]},{"osd":5,"uuid":"490a6258-1cf1-4c00-b819-c56c075b6c54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6809","nonce":542171958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6811","nonce":542171958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6815","nonce":542171958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6813","nonce":542171958}]},"public_addr":"172.21.15.172:6809/542171958","cluster_addr":"172.21.15.172:6811/542171958","heartbeat_back_addr":"172.21.15.172:6815/542171958","heartbeat_front_addr":"172.21.15.172:6813/542171958","state":["exists","up"]},{"osd":6,"uuid":"351cdd76-9bac-464a-84ff-ed1841688ea0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6817","nonce":2944243465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6819","nonce":2944243465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6823","nonce":2944243465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6821","nonce":2944243465}]},"public_addr":"172.21.15.172:6817/2944243465","cluster_addr":"172.21.15.172:6819/2944243465","heartbeat_back_addr":"172.21.15.172:6823/2944243465","heartbeat_front_addr":"172.21.15.172:6821/2944243465","state":["exists","up"]},{"osd":7,"uuid":"53d58400-5ee6-4fdb-87a3-6f10b10c8c7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6825","nonce":3721092024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6827","nonce":3721092024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6831","nonce":3721092024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6829","nonce":3721092024}]},"public_addr":"172.21.15.172:6825/3721092024","cluster_addr":"172.21.15.172:6827/3721092024","heartbeat_back_addr":"172.21.15.172:6831/3721092024","heartbeat_front_addr":"172.21.15.172:6829/3721092024","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:31:36.162215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:32:06.946929+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:32:33.223740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:32:58.600373+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:33:20.542112+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:33:44.501084+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:34:09.141620+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:34:33.864057+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:0/987860463":"2024-04-23T21:30:29.546191+0000","172.21.15.83:0/1419744190":"2024-04-23T21:30:29.546191+0000","172.21.15.83:6800/1063515625":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/4011448407":"2024-04-23T21:30:29.546191+0000","172.21.15.83:0/3651180963":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/3206574196":"2024-04-23T21:29:41.896421+0000","172.21.15.83:6801/121189037":"2024-04-23T21:30:29.546191+0000","172.21.15.83:6801/1063515625":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/414614811":"2024-04-23T21:29:12.027949+0000","172.21.15.83:6800/121189037":"2024-04-23T21:30:29.546191+0000","172.21.15.83:6801/29670807":"2024-04-23T21:29:12.027949+0000","172.21.15.83:0/3198040173":"2024-04-23T21:29:12.027949+0000","172.21.15.83:0/1929740090":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/4155452649":"2024-04-23T21:29:12.027949+0000","172.21.15.83:6800/29670807":"2024-04-23T21:29:12.027949+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T21:34:49.374 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:49 smithi083 ceph-mon[21788]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T21:34:49.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:49 smithi172 ceph-mon[23940]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T21:34:50.017 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T21:34:50.017 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T21:34:50.228 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:50.344 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:50 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1487876811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:34:50.470 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:50 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/1487876811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:34:51.517 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:51 smithi083 ceph-mon[21788]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s, 0 objects/s recovering 2024-04-22T21:34:51.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:51 smithi172 ceph-mon[23940]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s, 0 objects/s recovering 2024-04-22T21:34:52.024 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:34:52.024 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":59,"fsid":"3a5a1c14-00ef-11ef-bc93-c7b262605968","created":"2024-04-22T21:28:48.803761+0000","modified":"2024-04-22T21:34:39.189348+0000","last_up_change":"2024-04-22T21:34:35.663841+0000","last_in_change":"2024-04-22T21:34:18.124697+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T21:32:34.853617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"120f52f4-cd77-4920-9b7f-9e02e652e3e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6803","nonce":2774972645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6805","nonce":2774972645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6809","nonce":2774972645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2774972645},{"type":"v1","addr":"172.21.15.83:6807","nonce":2774972645}]},"public_addr":"172.21.15.83:6803/2774972645","cluster_addr":"172.21.15.83:6805/2774972645","heartbeat_back_addr":"172.21.15.83:6809/2774972645","heartbeat_front_addr":"172.21.15.83:6807/2774972645","state":["exists","up"]},{"osd":1,"uuid":"e885f222-674f-4a38-83bb-ab008b20607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6811","nonce":2475641293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6813","nonce":2475641293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6817","nonce":2475641293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":2475641293},{"type":"v1","addr":"172.21.15.83:6815","nonce":2475641293}]},"public_addr":"172.21.15.83:6811/2475641293","cluster_addr":"172.21.15.83:6813/2475641293","heartbeat_back_addr":"172.21.15.83:6817/2475641293","heartbeat_front_addr":"172.21.15.83:6815/2475641293","state":["exists","up"]},{"osd":2,"uuid":"ad91da27-115e-4ffc-8885-a6eb7fd412bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6819","nonce":2099285298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6821","nonce":2099285298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6825","nonce":2099285298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2099285298},{"type":"v1","addr":"172.21.15.83:6823","nonce":2099285298}]},"public_addr":"172.21.15.83:6819/2099285298","cluster_addr":"172.21.15.83:6821/2099285298","heartbeat_back_addr":"172.21.15.83:6825/2099285298","heartbeat_front_addr":"172.21.15.83:6823/2099285298","state":["exists","up"]},{"osd":3,"uuid":"939a71f6-56fd-4b75-b0ec-58d7ba5deb3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6827","nonce":678966071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6829","nonce":678966071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6833","nonce":678966071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":678966071},{"type":"v1","addr":"172.21.15.83:6831","nonce":678966071}]},"public_addr":"172.21.15.83:6827/678966071","cluster_addr":"172.21.15.83:6829/678966071","heartbeat_back_addr":"172.21.15.83:6833/678966071","heartbeat_front_addr":"172.21.15.83:6831/678966071","state":["exists","up"]},{"osd":4,"uuid":"c67d9990-dd06-4746-a651-443fb865e423","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6801","nonce":2779155763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6803","nonce":2779155763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6807","nonce":2779155763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":2779155763},{"type":"v1","addr":"172.21.15.172:6805","nonce":2779155763}]},"public_addr":"172.21.15.172:6801/2779155763","cluster_addr":"172.21.15.172:6803/2779155763","heartbeat_back_addr":"172.21.15.172:6807/2779155763","heartbeat_front_addr":"172.21.15.172:6805/2779155763","state":["exists","up"]},{"osd":5,"uuid":"490a6258-1cf1-4c00-b819-c56c075b6c54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6809","nonce":542171958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6811","nonce":542171958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6815","nonce":542171958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":542171958},{"type":"v1","addr":"172.21.15.172:6813","nonce":542171958}]},"public_addr":"172.21.15.172:6809/542171958","cluster_addr":"172.21.15.172:6811/542171958","heartbeat_back_addr":"172.21.15.172:6815/542171958","heartbeat_front_addr":"172.21.15.172:6813/542171958","state":["exists","up"]},{"osd":6,"uuid":"351cdd76-9bac-464a-84ff-ed1841688ea0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6817","nonce":2944243465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6819","nonce":2944243465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6823","nonce":2944243465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":2944243465},{"type":"v1","addr":"172.21.15.172:6821","nonce":2944243465}]},"public_addr":"172.21.15.172:6817/2944243465","cluster_addr":"172.21.15.172:6819/2944243465","heartbeat_back_addr":"172.21.15.172:6823/2944243465","heartbeat_front_addr":"172.21.15.172:6821/2944243465","state":["exists","up"]},{"osd":7,"uuid":"53d58400-5ee6-4fdb-87a3-6f10b10c8c7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6825","nonce":3721092024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6827","nonce":3721092024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6831","nonce":3721092024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3721092024},{"type":"v1","addr":"172.21.15.172:6829","nonce":3721092024}]},"public_addr":"172.21.15.172:6825/3721092024","cluster_addr":"172.21.15.172:6827/3721092024","heartbeat_back_addr":"172.21.15.172:6831/3721092024","heartbeat_front_addr":"172.21.15.172:6829/3721092024","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:31:36.162215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:32:06.946929+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:32:33.223740+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:32:58.600373+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:33:20.542112+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:33:44.501084+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:34:09.141620+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:34:33.864057+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:0/987860463":"2024-04-23T21:30:29.546191+0000","172.21.15.83:0/1419744190":"2024-04-23T21:30:29.546191+0000","172.21.15.83:6800/1063515625":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/4011448407":"2024-04-23T21:30:29.546191+0000","172.21.15.83:0/3651180963":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/3206574196":"2024-04-23T21:29:41.896421+0000","172.21.15.83:6801/121189037":"2024-04-23T21:30:29.546191+0000","172.21.15.83:6801/1063515625":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/414614811":"2024-04-23T21:29:12.027949+0000","172.21.15.83:6800/121189037":"2024-04-23T21:30:29.546191+0000","172.21.15.83:6801/29670807":"2024-04-23T21:29:12.027949+0000","172.21.15.83:0/3198040173":"2024-04-23T21:29:12.027949+0000","172.21.15.83:0/1929740090":"2024-04-23T21:29:41.896421+0000","172.21.15.83:0/4155452649":"2024-04-23T21:29:12.027949+0000","172.21.15.83:6800/29670807":"2024-04-23T21:29:12.027949+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T21:34:52.407 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:52 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/505291079' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:34:52.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:52 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/505291079' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:34:52.708 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T21:34:52.709 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T21:34:52.709 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T21:34:52.710 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T21:34:52.710 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T21:34:52.710 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T21:34:52.711 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-22T21:34:52.711 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-22T21:34:53.126 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:53.130 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:53.131 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:53.132 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:53.134 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:53.139 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:34:53.475 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:53 smithi083 ceph-mon[21788]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T21:34:53.495 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:53 smithi172 ceph-mon[23940]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T21:34:54.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:54 smithi083 ceph-mon[21788]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:34:55.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:54 smithi172 ceph-mon[23940]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:34:56.809 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:56 smithi083 ceph-mon[21788]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:34:57.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:56 smithi172 ceph-mon[23940]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:34:59.475 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:34:59 smithi083 ceph-mon[21788]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:34:59.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:34:59 smithi172 ceph-mon[23940]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:00.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:00 smithi172 ceph-mon[23940]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:00.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:00 smithi083 ceph-mon[21788]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:01.233 INFO:teuthology.orchestra.run.smithi083.stdout:236223201288 2024-04-22T21:35:01.233 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-22T21:35:02.700 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:35:02.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:02 smithi083 ceph-mon[21788]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:03.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:02 smithi172 ceph-mon[23940]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:04.771 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:04 smithi083 ceph-mon[21788]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:05.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:04 smithi172 ceph-mon[23940]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:05.638 INFO:teuthology.orchestra.run.smithi083.stdout:60129542182 2024-04-22T21:35:05.639 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T21:35:06.747 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:06 smithi083 ceph-mon[21788]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:06.773 INFO:teuthology.orchestra.run.smithi083.stdout:201863462926 2024-04-22T21:35:06.774 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-22T21:35:07.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:06 smithi172 ceph-mon[23940]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:07.163 INFO:teuthology.orchestra.run.smithi083.stdout:167503724562 2024-04-22T21:35:07.163 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T21:35:07.918 INFO:teuthology.orchestra.run.smithi083.stdout:38654705708 2024-04-22T21:35:07.918 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T21:35:09.362 INFO:teuthology.orchestra.run.smithi083.stdout:81604378657 2024-04-22T21:35:09.362 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T21:35:09.473 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:09 smithi083 ceph-mon[21788]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:09.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:09 smithi172 ceph-mon[23940]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:10.207 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:35:10.620 INFO:teuthology.orchestra.run.smithi083.stdout:111669149725 2024-04-22T21:35:10.621 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T21:35:10.826 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:35:10.830 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:35:10.830 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:35:10.830 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:35:10.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:10 smithi172 ceph-mon[23940]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:10.888 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:35:10.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:10 smithi083 ceph-mon[21788]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:12.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:12 smithi083 ceph-mon[21788]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:13.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:12 smithi172 ceph-mon[23940]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:14.312 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:14 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1307566448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T21:35:14.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:14 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/1307566448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T21:35:14.633 INFO:teuthology.orchestra.run.smithi083.stdout:236223201291 2024-04-22T21:35:15.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:15 smithi172 ceph-mon[23940]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:15.639 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:15 smithi083 ceph-mon[21788]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:15.667 INFO:teuthology.orchestra.run.smithi083.stdout:133143986200 2024-04-22T21:35:15.668 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T21:35:16.461 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201288 got 236223201291 for osd.7 2024-04-22T21:35:16.461 DEBUG:teuthology.parallel:result is None 2024-04-22T21:35:16.707 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:16 smithi083 ceph-mon[21788]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:17.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:16 smithi172 ceph-mon[23940]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:18.129 INFO:teuthology.orchestra.run.smithi083.stdout:201863462928 2024-04-22T21:35:18.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:18 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1316253994' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T21:35:18.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:18 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/1316253994' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T21:35:18.868 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462926 got 201863462928 for osd.6 2024-04-22T21:35:18.869 DEBUG:teuthology.parallel:result is None 2024-04-22T21:35:19.048 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:35:19.240 INFO:teuthology.orchestra.run.smithi083.stdout:167503724565 2024-04-22T21:35:19.539 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:19 smithi083 ceph-mon[21788]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:19.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:19 smithi172 ceph-mon[23940]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:19.889 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724562 got 167503724565 for osd.5 2024-04-22T21:35:19.889 DEBUG:teuthology.parallel:result is None 2024-04-22T21:35:20.470 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:20 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/3081660614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T21:35:20.575 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:20 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3081660614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T21:35:21.019 INFO:teuthology.orchestra.run.smithi083.stdout:60129542185 2024-04-22T21:35:21.337 INFO:teuthology.orchestra.run.smithi083.stdout:81604378661 2024-04-22T21:35:21.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:21 smithi083 ceph-mon[21788]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:21.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:21 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3102912575' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T21:35:21.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:21 smithi172 ceph-mon[23940]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:21.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:21 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/3102912575' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T21:35:22.360 INFO:teuthology.orchestra.run.smithi083.stdout:133143986202 2024-04-22T21:35:22.360 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:22 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1206686204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T21:35:22.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:22 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/1206686204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T21:35:23.377 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:23 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/789387332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T21:35:23.377 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:23 smithi083 ceph-mon[21788]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:23.495 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:23 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/789387332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T21:35:23.496 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:23 smithi172 ceph-mon[23940]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:23.731 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986200 got 133143986202 for osd.4 2024-04-22T21:35:23.731 DEBUG:teuthology.parallel:result is None 2024-04-22T21:35:24.103 INFO:teuthology.orchestra.run.smithi083.stdout:38654705711 2024-04-22T21:35:24.307 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542182 got 60129542185 for osd.1 2024-04-22T21:35:24.308 DEBUG:teuthology.parallel:result is None 2024-04-22T21:35:24.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:24 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/2651318636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T21:35:24.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:24 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/2651318636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T21:35:24.944 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378657 got 81604378661 for osd.2 2024-04-22T21:35:24.945 DEBUG:teuthology.parallel:result is None 2024-04-22T21:35:25.505 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:25 smithi083 ceph-mon[21788]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:25.587 INFO:teuthology.orchestra.run.smithi083.stdout:111669149728 2024-04-22T21:35:25.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:25 smithi172 ceph-mon[23940]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:26.011 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705708 got 38654705711 for osd.0 2024-04-22T21:35:26.011 DEBUG:teuthology.parallel:result is None 2024-04-22T21:35:26.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:26 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1907004115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T21:35:26.430 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149725 got 111669149728 for osd.3 2024-04-22T21:35:26.430 DEBUG:teuthology.parallel:result is None 2024-04-22T21:35:26.430 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T21:35:26.430 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T21:35:26.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:26 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/1907004115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T21:35:26.641 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:35:27.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:27 smithi083 ceph-mon[21788]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:27.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:27 smithi172 ceph-mon[23940]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:28.298 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:35:28.300 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-04-22T21:35:28.895 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-04-22T21:35:26.594371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48748,"kb_used_data":6316,"kb_used_omap":0,"kb_used_meta":42368,"kb_avail":749912468,"statfs":{"total":767960285184,"available":767910367232,"internally_reserved":0,"allocated":6467584,"data_stored":3547795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43384832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002549"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T21:34:39.195330+0000","last_change":"2024-04-22T21:34:39.195330+0000","last_active":"2024-04-22T21:34:39.195330+0000","last_peered":"2024-04-22T21:34:39.195330+0000","last_clean":"2024-04-22T21:34:39.195330+0000","last_became_active":"2024-04-22T21:34:39.194843+0000","last_became_peered":"2024-04-22T21:34:39.194843+0000","last_unstale":"2024-04-22T21:34:39.195330+0000","last_undegraded":"2024-04-22T21:34:39.195330+0000","last_fullsized":"2024-04-22T21:34:39.195330+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T21:32:35.316703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T21:32:35.316703+0000","last_clean_scrub_stamp":"2024-04-22T21:32:35.316703+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T02:42:21.655105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6048,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739104,"statfs":{"total":95995035648,"available":95988842496,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6112,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739040,"statfs":{"total":95995035648,"available":95988776960,"internally_reserved":0,"allocated":1138688,"data_stored":773896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.436},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.499}]},{"osd":2,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.45300000000000001}]},{"osd":3,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.55300000000000005}]},{"osd":4,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.58299999999999996}]},{"osd":5,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6360,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.30599999999999999},"min":{"1min":0.22,"5min":0.218,"15min":0.218},"max":{"1min":0.53900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.34899999999999998,"15min":0.32800000000000001},"min":{"1min":0.247,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Apr 22 21:34:42 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.35899999999999999,"15min":0.34699999999999998},"min":{"1min":0.29699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.36399999999999999,"15min":0.35799999999999998},"min":{"1min":0.309,"5min":0.247,"15min":0.247},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.309}]},{"osd":3,"last update":"Mon Apr 22 21:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55300000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.438,"15min":0.438},"min":{"1min":0.30099999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.29499999999999998}]},{"osd":4,"last update":"Mon Apr 22 21:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.39800000000000002}]},{"osd":5,"last update":"Mon Apr 22 21:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.432},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Apr 22 21:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.44700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705712,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 21:35:10 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.33400000000000002,"15min":0.29899999999999999},"min":{"1min":0.252,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.33300000000000002,"15min":0.30199999999999999},"min":{"1min":0.27800000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.497}]},{"osd":2,"last update":"Mon Apr 22 21:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.34999999999999998,"15min":0.34100000000000003},"min":{"1min":0.26600000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.35999999999999999,"15min":0.35099999999999998},"min":{"1min":0.32500000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.54100000000000004}]},{"osd":3,"last update":"Mon Apr 22 21:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41999999999999998,"15min":0.41499999999999998},"min":{"1min":0.34000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.47299999999999998}]},{"osd":4,"last update":"Mon Apr 22 21:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.67000000000000004}]},{"osd":5,"last update":"Mon Apr 22 21:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Apr 22 21:35:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.72199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6236,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35399999999999998,"15min":0.35299999999999998},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.25},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.376,"15min":0.372},"min":{"1min":0.251,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.20000000000000001}]},{"osd":1,"last update":"Mon Apr 22 21:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.377,"15min":0.371},"min":{"1min":0.32500000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.23999999999999999}]},{"osd":3,"last update":"Mon Apr 22 21:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.60599999999999998,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59199999999999997,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.26200000000000001}]},{"osd":4,"last update":"Mon Apr 22 21:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.59699999999999998}]},{"osd":5,"last update":"Mon Apr 22 21:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.66500000000000004}]},{"osd":6,"last update":"Mon Apr 22 21:35:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.61299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.41399999999999998,"15min":0.42399999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60699999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.26200000000000001}]},{"osd":1,"last update":"Mon Apr 22 21:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.443,"15min":0.45200000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.66000000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.316},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.442,"15min":0.45000000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63100000000000001,"5min":0.745,"15min":0.745},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Apr 22 21:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.46400000000000002,"15min":0.47399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.35199999999999998}]},{"osd":4,"last update":"Mon Apr 22 21:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Apr 22 21:34:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Apr 22 21:35:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.67800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.51200000000000001}]},{"osd":1,"last update":"Mon Apr 22 21:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.498},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Apr 22 21:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.44},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.41699999999999998}]},{"osd":3,"last update":"Mon Apr 22 21:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.52700000000000002}]},{"osd":5,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.372},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Apr 22 21:35:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.57899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724567,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5660,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739492,"statfs":{"total":95995035648,"available":95989239808,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.59099999999999997}]},{"osd":2,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.51800000000000002}]},{"osd":3,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.624},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.56699999999999995}]},{"osd":4,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T21:35:28.896 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T21:35:29.105 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:35:29.499 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:29 smithi083 ceph-mon[21788]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:35:29.499 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:29 smithi083 ceph-mon[21788]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:29.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:29 smithi172 ceph-mon[23940]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:35:29.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:29 smithi172 ceph-mon[23940]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:30.666 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:35:30.668 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-04-22T21:35:30.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:30 smithi172 ceph-mon[23940]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:30.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:35:30.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:35:30.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:30 smithi083 ceph-mon[21788]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:30.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:35:30.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:35:31.282 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-22T21:35:30.594948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48748,"kb_used_data":6316,"kb_used_omap":0,"kb_used_meta":42368,"kb_avail":749912468,"statfs":{"total":767960285184,"available":767910367232,"internally_reserved":0,"allocated":6467584,"data_stored":3547795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43384832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002181"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T21:34:39.195330+0000","last_change":"2024-04-22T21:34:39.195330+0000","last_active":"2024-04-22T21:34:39.195330+0000","last_peered":"2024-04-22T21:34:39.195330+0000","last_clean":"2024-04-22T21:34:39.195330+0000","last_became_active":"2024-04-22T21:34:39.194843+0000","last_became_peered":"2024-04-22T21:34:39.194843+0000","last_unstale":"2024-04-22T21:34:39.195330+0000","last_undegraded":"2024-04-22T21:34:39.195330+0000","last_fullsized":"2024-04-22T21:34:39.195330+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T21:32:35.316703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T21:32:35.316703+0000","last_clean_scrub_stamp":"2024-04-22T21:32:35.316703+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T02:42:21.655105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6048,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739104,"statfs":{"total":95995035648,"available":95988842496,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]}]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6112,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739040,"statfs":{"total":95995035648,"available":95988776960,"internally_reserved":0,"allocated":1138688,"data_stored":773896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.58799999999999997}]},{"osd":1,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.64500000000000002}]},{"osd":5,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6360,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.30599999999999999},"min":{"1min":0.22,"5min":0.218,"15min":0.218},"max":{"1min":0.53900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.316},{"interface":"front","average":{"1min":0.377,"5min":0.34899999999999998,"15min":0.32800000000000001},"min":{"1min":0.247,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.22800000000000001}]},{"osd":2,"last update":"Mon Apr 22 21:34:42 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.35899999999999999,"15min":0.34699999999999998},"min":{"1min":0.29699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.36399999999999999,"15min":0.35799999999999998},"min":{"1min":0.309,"5min":0.247,"15min":0.247},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.254}]},{"osd":3,"last update":"Mon Apr 22 21:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55300000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.438,"15min":0.438},"min":{"1min":0.30099999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Apr 22 21:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.41999999999999998}]},{"osd":5,"last update":"Mon Apr 22 21:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.5}]},{"osd":6,"last update":"Mon Apr 22 21:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.39300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 21:35:10 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.33400000000000002,"15min":0.29899999999999999},"min":{"1min":0.252,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.33300000000000002,"15min":0.30199999999999999},"min":{"1min":0.27800000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.434}]},{"osd":2,"last update":"Mon Apr 22 21:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.34999999999999998,"15min":0.34100000000000003},"min":{"1min":0.26600000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.35999999999999999,"15min":0.35099999999999998},"min":{"1min":0.32500000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Apr 22 21:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41999999999999998,"15min":0.41499999999999998},"min":{"1min":0.34000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.45700000000000002}]},{"osd":4,"last update":"Mon Apr 22 21:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.53300000000000003,"15min":0.51600000000000001},"min":{"1min":0.40999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.498,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.69599999999999995}]},{"osd":5,"last update":"Mon Apr 22 21:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.752},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.74199999999999999}]},{"osd":6,"last update":"Mon Apr 22 21:35:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6236,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35399999999999998,"15min":0.35299999999999998},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.376,"15min":0.372},"min":{"1min":0.251,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.372}]},{"osd":1,"last update":"Mon Apr 22 21:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.434},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.377,"15min":0.371},"min":{"1min":0.32500000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.41999999999999998}]},{"osd":3,"last update":"Mon Apr 22 21:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.60599999999999998,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59199999999999997,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.44900000000000001}]},{"osd":4,"last update":"Mon Apr 22 21:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.40699999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67500000000000004,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.48399999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.68999999999999995,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.70199999999999996}]},{"osd":5,"last update":"Mon Apr 22 21:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.79300000000000004}]},{"osd":6,"last update":"Mon Apr 22 21:35:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.84799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.41399999999999998,"15min":0.42399999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60699999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.26200000000000001}]},{"osd":1,"last update":"Mon Apr 22 21:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.443,"15min":0.45200000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.66000000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.316},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.442,"15min":0.45000000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63100000000000001,"5min":0.745,"15min":0.745},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Apr 22 21:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.46400000000000002,"15min":0.47399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.35199999999999998}]},{"osd":4,"last update":"Mon Apr 22 21:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Apr 22 21:34:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Apr 22 21:35:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.67800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.57699999999999996,"15min":0.58699999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.58999999999999997,"15min":0.60699999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69699999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.60299999999999998}]},{"osd":1,"last update":"Mon Apr 22 21:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78900000000000003,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68799999999999994,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Apr 22 21:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.58799999999999997,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.751,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.58699999999999997,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.63600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Apr 22 21:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.622,"15min":0.63500000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67400000000000004,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.625},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.58299999999999996,"15min":0.58999999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.45500000000000002}]},{"osd":5,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Apr 22 21:35:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":5,"up_from":39,"seq":167503724567,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5660,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739492,"statfs":{"total":95995035648,"available":95989239808,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.59099999999999997}]},{"osd":2,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.51800000000000002}]},{"osd":3,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.624},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.56699999999999995}]},{"osd":4,"last update":"Mon Apr 22 21:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Apr 22 21:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T21:35:31.283 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T21:35:31.284 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T21:35:31.284 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T21:35:31.284 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T21:35:31.495 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/mon.smithi083/config 2024-04-22T21:35:31.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:31 smithi083 ceph-mon[21788]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:35:32.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:31 smithi172 ceph-mon[23940]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:35:32.699 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:32 smithi083 ceph-mon[21788]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:33.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:32 smithi172 ceph-mon[23940]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:33.355 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:35:33.355 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T21:35:33.970 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T21:35:33.970 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T21:35:33.970 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:35:33.979 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi083.front.sepia.ceph.com 2024-04-22T21:35:33.979 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T21:35:34.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:34 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/1004780861' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T21:35:34.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:34 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/1004780861' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T21:35:35.499 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:35 smithi083 ceph-mon[21788]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:35.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:35 smithi172 ceph-mon[23940]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:35.872 INFO:teuthology.orchestra.run.smithi083.stdout:Backend: cephadm 2024-04-22T21:35:35.872 INFO:teuthology.orchestra.run.smithi083.stdout:Available: Yes 2024-04-22T21:35:35.872 INFO:teuthology.orchestra.run.smithi083.stdout:Paused: No 2024-04-22T21:35:36.467 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T21:35:36.898 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:36 smithi083 ceph-mon[21788]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:36.898 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:36 smithi083 ceph-mon[21788]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:37.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:36 smithi172 ceph-mon[23940]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:37.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:36 smithi172 ceph-mon[23940]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:37.767 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:37 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:37.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:37 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:37.768 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:37 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:37.779 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:37 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:37.779 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:37 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:37.779 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:37 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:38.446 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T21:35:38.446 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.smithi083 smithi083 *:9093,9094 running (4m) 2m ago 5m 13.8M - ba2b418f427c b40e5826cf4a 2024-04-22T21:35:38.446 INFO:teuthology.orchestra.run.smithi083.stdout:crash.smithi083 smithi083 running (5m) 2m ago 5m 6409k - 17.2.0 e1d6a67b021e 8017849c4e67 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:crash.smithi172 smithi172 running (4m) 64s ago 4m 6409k - 17.2.0 e1d6a67b021e 579757f6e520 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.smithi083 smithi083 *:3000 running (4m) 2m ago 5m 42.2M - 8.3.5 dad864ee21e9 e6c664d97098 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.smithi083.ochghz smithi083 *:9283 running (6m) 2m ago 6m 448M - 17.2.0 e1d6a67b021e e6931af0de17 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.smithi172.edmjti smithi172 *:8443,9283 running (4m) 64s ago 4m 411M - 17.2.0 e1d6a67b021e 35ff72a3ae30 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:mon.smithi083 smithi083 running (6m) 2m ago 6m 45.0M 2048M 17.2.0 e1d6a67b021e 25f27fe2e2e7 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:mon.smithi172 smithi172 running (4m) 64s ago 4m 39.3M 2048M 17.2.0 e1d6a67b021e 2c168ec547f8 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.smithi083 smithi083 *:9100 running (5m) 2m ago 5m 18.6M - 1dbe0e931976 72d59242559d 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.smithi172 smithi172 *:9100 running (4m) 64s ago 4m 17.5M - 1dbe0e931976 111fb1992476 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (4m) 2m ago 4m 32.3M 3216M 17.2.0 e1d6a67b021e f33f10a14180 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (3m) 2m ago 3m 33.8M 3216M 17.2.0 e1d6a67b021e fe80eaef93c7 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (3m) 2m ago 3m 31.2M 3216M 17.2.0 e1d6a67b021e cb42fed21ebe 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (2m) 2m ago 2m 10.2M 3216M 17.2.0 e1d6a67b021e f5943d14bc30 2024-04-22T21:35:38.447 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi172 running (2m) 64s ago 2m 32.8M 3984M 17.2.0 e1d6a67b021e 2627d45f1040 2024-04-22T21:35:38.448 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi172 running (118s) 64s ago 118s 33.3M 3984M 17.2.0 e1d6a67b021e 6bd4a960e7d5 2024-04-22T21:35:38.448 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi172 running (93s) 64s ago 93s 30.0M 3984M 17.2.0 e1d6a67b021e c5e86086662c 2024-04-22T21:35:38.448 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi172 running (69s) 64s ago 68s 10.9M 3984M 17.2.0 e1d6a67b021e 80378e864f5e 2024-04-22T21:35:38.448 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.smithi083 smithi083 *:9095 running (4m) 2m ago 4m 51.9M - 514e6a882f6e d268cb8349fb 2024-04-22T21:35:39.066 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:38 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:39.066 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:38 smithi083 ceph-mon[21788]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:39.067 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:38 smithi083 ceph-mon[21788]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:39.089 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T21:35:39.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:38 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:39.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:38 smithi172 ceph-mon[23940]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:39.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:38 smithi172 ceph-mon[23940]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:40.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:40 smithi172 ceph-mon[23940]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:40.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:40 smithi083 ceph-mon[21788]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:40.992 INFO:teuthology.orchestra.run.smithi083.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T21:35:40.992 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-22T21:35:40.992 INFO:teuthology.orchestra.run.smithi083.stdout:crash 2/2 2m ago 6m * 2024-04-22T21:35:40.992 INFO:teuthology.orchestra.run.smithi083.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-22T21:35:40.992 INFO:teuthology.orchestra.run.smithi083.stdout:mgr 2/2 2m ago 6m count:2 2024-04-22T21:35:40.992 INFO:teuthology.orchestra.run.smithi083.stdout:mon 2/2 2m ago 5m smithi083:172.21.15.83=smithi083;smithi172:172.21.15.172=smithi172;count:2 2024-04-22T21:35:40.993 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-22T21:35:40.993 INFO:teuthology.orchestra.run.smithi083.stdout:osd 8 2m ago - 2024-04-22T21:35:40.993 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-22T21:35:41.684 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T21:35:41.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:41 smithi083 ceph-mon[21788]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:42.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:41 smithi172 ceph-mon[23940]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:42.947 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:42 smithi083 ceph-mon[21788]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:43.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:42 smithi172 ceph-mon[23940]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:43.615 INFO:teuthology.orchestra.run.smithi083.stdout:HOST ADDR LABELS STATUS 2024-04-22T21:35:43.615 INFO:teuthology.orchestra.run.smithi083.stdout:smithi083 172.21.15.83 2024-04-22T21:35:43.615 INFO:teuthology.orchestra.run.smithi083.stdout:smithi172 172.21.15.172 2024-04-22T21:35:43.615 INFO:teuthology.orchestra.run.smithi083.stdout:2 hosts in cluster 2024-04-22T21:35:44.204 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T21:35:44.825 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:44 smithi083 ceph-mon[21788]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:44.825 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:44 smithi083 ceph-mon[21788]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:45.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:44 smithi172 ceph-mon[23940]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:45.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:44 smithi172 ceph-mon[23940]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:46.142 INFO:teuthology.orchestra.run.smithi083.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-04-22T21:35:46.142 INFO:teuthology.orchestra.run.smithi083.stdout:smithi083 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040148400BGN 400G LVM detected, locked 2024-04-22T21:35:46.142 INFO:teuthology.orchestra.run.smithi083.stdout:smithi172 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233005M400BGN 400G LVM detected, locked 2024-04-22T21:35:46.760 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-22T21:35:46.772 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi083.front.sepia.ceph.com 2024-04-22T21:35:46.773 DEBUG:teuthology.orchestra.run.smithi083:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-22T21:35:46.799 INFO:teuthology.orchestra.run.smithi083.stderr:+ systemctl stop nfs-server 2024-04-22T21:35:46.806 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T21:35:46.806 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-22T21:35:46.835 INFO:teuthology.orchestra.run.smithi172.stderr:+ systemctl stop nfs-server 2024-04-22T21:35:46.843 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:35:46.854 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi083.front.sepia.ceph.com 2024-04-22T21:35:46.854 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- bash -c 'ceph fs volume create foofs' 2024-04-22T21:35:46.875 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:46 smithi083 ceph-mon[21788]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:46.875 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:46 smithi083 ceph-mon[21788]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:47.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:46 smithi172 ceph-mon[23940]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:47.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:46 smithi172 ceph-mon[23940]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:49.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:49 smithi172 ceph-mon[23940]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:49.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:49 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-22T21:35:49.649 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:49 smithi083 ceph-mon[21788]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:49.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:49 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-22T21:35:50.469 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:50 smithi172 ceph-mon[23940]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:50.470 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-22T21:35:50.470 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:50 smithi172 ceph-mon[23940]: osdmap e60: 8 total, 8 up, 8 in 2024-04-22T21:35:50.470 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-22T21:35:50.599 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:50 smithi083 ceph-mon[21788]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:50.599 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-22T21:35:50.599 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:50 smithi083 ceph-mon[21788]: osdmap e60: 8 total, 8 up, 8 in 2024-04-22T21:35:50.599 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-22T21:35:50.599 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:50 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083[21784]: 2024-04-22T21:35:50.215+0000 7f62e0291700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T21:35:50.897 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-04-22T21:35:50.911 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-04-22T21:35:50.912 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-22T21:35:51.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: osdmap e61: 8 total, 8 up, 8 in 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: osdmap e62: 8 total, 8 up, 8 in 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: fsmap foofs:0 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: Saving service mds.foofs spec with placement count:2 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:51.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:51.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi083.nyheuj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:35:51.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi083.nyheuj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:35:51.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:51.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: Deploying daemon mds.foofs.smithi083.nyheuj on smithi083 2024-04-22T21:35:51.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:51 smithi172 ceph-mon[23940]: pgmap v210: 65 pgs: 8 creating+peering, 10 active+clean, 47 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:51.649 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-22T21:35:51.649 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: osdmap e61: 8 total, 8 up, 8 in 2024-04-22T21:35:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-22T21:35:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T21:35:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T21:35:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: osdmap e62: 8 total, 8 up, 8 in 2024-04-22T21:35:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-22T21:35:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: fsmap foofs:0 2024-04-22T21:35:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: Saving service mds.foofs spec with placement count:2 2024-04-22T21:35:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:51.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi083.nyheuj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:35:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi083.nyheuj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:35:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: Deploying daemon mds.foofs.smithi083.nyheuj on smithi083 2024-04-22T21:35:51.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:51 smithi083 ceph-mon[21788]: pgmap v210: 65 pgs: 8 creating+peering, 10 active+clean, 47 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:52.472 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:52 smithi083 ceph-mon[21788]: osdmap e63: 8 total, 8 up, 8 in 2024-04-22T21:35:52.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:52 smithi172 ceph-mon[23940]: osdmap e63: 8 total, 8 up, 8 in 2024-04-22T21:35:53.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:53 smithi083 ceph-mon[21788]: osdmap e64: 8 total, 8 up, 8 in 2024-04-22T21:35:53.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:53 smithi083 ceph-mon[21788]: pgmap v213: 65 pgs: 13 creating+peering, 36 active+clean, 16 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:53.495 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:53 smithi172 ceph-mon[23940]: osdmap e64: 8 total, 8 up, 8 in 2024-04-22T21:35:53.495 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:53 smithi172 ceph-mon[23940]: pgmap v213: 65 pgs: 13 creating+peering, 36 active+clean, 16 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:53.687 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:35:53.687 INFO:teuthology.orchestra.run.smithi083.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-22T21:29:31.324140Z", "last_refresh": "2024-04-22T21:33:01.310866Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-22T21:29:25.020976Z", "last_refresh": "2024-04-22T21:33:01.310930Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-22T21:29:28.831410Z", "last_refresh": "2024-04-22T21:33:01.310992Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-22T21:35:50.234654Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-22T21:35:50.229488Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-22T21:29:22.869721Z", "last_refresh": "2024-04-22T21:33:01.310795Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi083:172.21.15.83=smithi083", "smithi172:172.21.15.172=smithi172"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-22T21:30:13.758556Z", "last_refresh": "2024-04-22T21:33:01.310656Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-22T21:29:30.051253Z", "last_refresh": "2024-04-22T21:33:01.311066Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-22T21:33:01.311189Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-22T21:29:26.755220Z", "last_refresh": "2024-04-22T21:33:01.311129Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-22T21:35:54.370 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-22T21:35:55.371 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-22T21:35:55.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:35:55.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: pgmap v214: 65 pgs: 13 creating+peering, 49 active+clean, 3 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:55.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: mds.? [v2:172.21.15.83:6834/4118377244,v1:172.21.15.83:6835/4118377244] up:boot 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: daemon mds.foofs.smithi083.nyheuj assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: Cluster is now healthy 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: fsmap foofs:0 1 up:standby 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi083.nyheuj"}]: dispatch 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: fsmap foofs:1 {0=foofs.smithi083.nyheuj=up:creating} 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: daemon mds.foofs.smithi083.nyheuj is now active in filesystem foofs as rank 0 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi172.cavhbk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi172.cavhbk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:35:55.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:55 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:55.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:35:55.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: pgmap v214: 65 pgs: 13 creating+peering, 49 active+clean, 3 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:35:55.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: mds.? [v2:172.21.15.83:6834/4118377244,v1:172.21.15.83:6835/4118377244] up:boot 2024-04-22T21:35:55.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: daemon mds.foofs.smithi083.nyheuj assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-22T21:35:55.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T21:35:55.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T21:35:55.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: Cluster is now healthy 2024-04-22T21:35:55.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: fsmap foofs:0 1 up:standby 2024-04-22T21:35:55.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi083.nyheuj"}]: dispatch 2024-04-22T21:35:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: fsmap foofs:1 {0=foofs.smithi083.nyheuj=up:creating} 2024-04-22T21:35:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: daemon mds.foofs.smithi083.nyheuj is now active in filesystem foofs as rank 0 2024-04-22T21:35:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi172.cavhbk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T21:35:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi172.cavhbk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T21:35:55.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:55 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:56.747 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:56 smithi172 ceph-mon[23940]: Deploying daemon mds.foofs.smithi172.cavhbk on smithi172 2024-04-22T21:35:56.747 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:56 smithi172 ceph-mon[23940]: mds.? [v2:172.21.15.83:6834/4118377244,v1:172.21.15.83:6835/4118377244] up:active 2024-04-22T21:35:56.747 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:56 smithi172 ceph-mon[23940]: fsmap foofs:1 {0=foofs.smithi083.nyheuj=up:active} 2024-04-22T21:35:56.865 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:56 smithi083 ceph-mon[21788]: Deploying daemon mds.foofs.smithi172.cavhbk on smithi172 2024-04-22T21:35:56.865 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:56 smithi083 ceph-mon[21788]: mds.? [v2:172.21.15.83:6834/4118377244,v1:172.21.15.83:6835/4118377244] up:active 2024-04-22T21:35:56.865 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:56 smithi083 ceph-mon[21788]: fsmap foofs:1 {0=foofs.smithi083.nyheuj=up:active} 2024-04-22T21:35:57.268 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:35:57.268 INFO:teuthology.orchestra.run.smithi083.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-22T21:29:31.324140Z", "last_refresh": "2024-04-22T21:33:01.310866Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-22T21:29:25.020976Z", "last_refresh": "2024-04-22T21:33:01.310930Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-22T21:29:28.831410Z", "last_refresh": "2024-04-22T21:33:01.310992Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-22T21:35:50.234654Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-22T21:35:50.229488Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-22T21:29:22.869721Z", "last_refresh": "2024-04-22T21:33:01.310795Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi083:172.21.15.83=smithi083", "smithi172:172.21.15.172=smithi172"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-22T21:30:13.758556Z", "last_refresh": "2024-04-22T21:33:01.310656Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-22T21:29:30.051253Z", "last_refresh": "2024-04-22T21:33:01.311066Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-22T21:33:01.311189Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-22T21:29:26.755220Z", "last_refresh": "2024-04-22T21:33:01.311129Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-22T21:35:57.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:57 smithi172 ceph-mon[23940]: pgmap v215: 65 pgs: 65 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 803 B/s wr, 4 op/s 2024-04-22T21:35:57.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:57 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:57.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:57 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:57.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:57 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:57.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:57 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:57.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:57 smithi083 ceph-mon[21788]: pgmap v215: 65 pgs: 65 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 803 B/s wr, 4 op/s 2024-04-22T21:35:57.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:57 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:57.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:57 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:57.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:57 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:57.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:57 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:58.036 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-22T21:35:58.649 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:58 smithi083 ceph-mon[21788]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:35:58.649 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:58 smithi083 ceph-mon[21788]: mds.? [v2:172.21.15.172:6832/3940959071,v1:172.21.15.172:6833/3940959071] up:boot 2024-04-22T21:35:58.649 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:58 smithi083 ceph-mon[21788]: fsmap foofs:1 {0=foofs.smithi083.nyheuj=up:active} 1 up:standby 2024-04-22T21:35:58.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:58 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi172.cavhbk"}]: dispatch 2024-04-22T21:35:58.650 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:58 smithi083 ceph-mon[21788]: fsmap foofs:1 {0=foofs.smithi083.nyheuj=up:active} 1 up:standby 2024-04-22T21:35:58.836 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:58 smithi172 ceph-mon[23940]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:35:58.837 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:58 smithi172 ceph-mon[23940]: mds.? [v2:172.21.15.172:6832/3940959071,v1:172.21.15.172:6833/3940959071] up:boot 2024-04-22T21:35:58.837 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:58 smithi172 ceph-mon[23940]: fsmap foofs:1 {0=foofs.smithi083.nyheuj=up:active} 1 up:standby 2024-04-22T21:35:58.837 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:58 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi172.cavhbk"}]: dispatch 2024-04-22T21:35:58.837 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:58 smithi172 ceph-mon[23940]: fsmap foofs:1 {0=foofs.smithi083.nyheuj=up:active} 1 up:standby 2024-04-22T21:35:59.037 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-22T21:35:59.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:59 smithi172 ceph-mon[23940]: pgmap v216: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-22T21:35:59.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:35:59 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:35:59.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:59 smithi083 ceph-mon[21788]: pgmap v216: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-22T21:35:59.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:35:59 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:00.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:00 smithi172 ceph-mon[23940]: pgmap v217: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-22T21:36:00.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:00 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:00.846 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:00 smithi083 ceph-mon[21788]: pgmap v217: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-22T21:36:00.846 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:00 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:02.552 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:36:02.552 INFO:teuthology.orchestra.run.smithi083.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-22T21:29:31.324140Z", "last_refresh": "2024-04-22T21:33:01.310866Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-22T21:29:25.020976Z", "last_refresh": "2024-04-22T21:33:01.310930Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-22T21:29:28.831410Z", "last_refresh": "2024-04-22T21:33:01.310992Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-22T21:35:50.234654Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-22T21:35:50.229488Z", "last_refresh": "2024-04-22T21:35:59.384108Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-22T21:29:22.869721Z", "last_refresh": "2024-04-22T21:33:01.310795Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi083:172.21.15.83=smithi083", "smithi172:172.21.15.172=smithi172"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-22T21:30:13.758556Z", "last_refresh": "2024-04-22T21:33:01.310656Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-22T21:29:30.051253Z", "last_refresh": "2024-04-22T21:33:01.311066Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-22T21:33:01.311189Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-22T21:29:26.755220Z", "last_refresh": "2024-04-22T21:33:01.311129Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-22T21:36:02.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:02 smithi083 ceph-mon[21788]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:36:02.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:02 smithi083 ceph-mon[21788]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-22T21:36:03.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:02 smithi172 ceph-mon[23940]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:36:03.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:02 smithi172 ceph-mon[23940]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-22T21:36:03.449 INFO:tasks.cephadm:mds.foofs has 1/2 2024-04-22T21:36:04.450 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-22T21:36:05.047 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:04 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:05.047 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:04 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:05.047 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:04 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:05.047 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:04 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:05.047 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:04 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:05.047 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:04 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:05.047 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:04 smithi083 ceph-mon[21788]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-22T21:36:05.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:04 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:05.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:04 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:05.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:04 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:05.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:04 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:05.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:04 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:05.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:04 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:05.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:04 smithi172 ceph-mon[23940]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-22T21:36:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:36:06.330 INFO:teuthology.orchestra.run.smithi083.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-22T21:29:31.324140Z", "last_refresh": "2024-04-22T21:36:03.968933Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-22T21:29:25.020976Z", "last_refresh": "2024-04-22T21:35:59.383530Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-22T21:29:28.831410Z", "last_refresh": "2024-04-22T21:36:03.969074Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-22T21:35:50.234654Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-22T21:35:50.229488Z", "last_refresh": "2024-04-22T21:35:59.384108Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-22T21:29:22.869721Z", "last_refresh": "2024-04-22T21:35:59.383670Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi083:172.21.15.83=smithi083", "smithi172:172.21.15.172=smithi172"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-22T21:30:13.758556Z", "last_refresh": "2024-04-22T21:35:59.383739Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-22T21:29:30.051253Z", "last_refresh": "2024-04-22T21:35:59.383801Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-22T21:35:59.383860Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-22T21:29:26.755220Z", "last_refresh": "2024-04-22T21:36:03.969227Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-22T21:36:06.865 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:06 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:06.865 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:06 smithi083 ceph-mon[21788]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:36:06.865 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:06 smithi083 ceph-mon[21788]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-22T21:36:06.888 INFO:tasks.cephadm:mds.foofs has 2/2 2024-04-22T21:36:06.889 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:36:06.899 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi083.front.sepia.ceph.com 2024-04-22T21:36:06.900 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-04-22T21:36:07.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:06 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:07.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:06 smithi172 ceph-mon[23940]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:36:07.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:06 smithi172 ceph-mon[23940]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-22T21:36:09.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:09 smithi172 ceph-mon[23940]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 1 op/s 2024-04-22T21:36:09.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:09 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-22T21:36:09.640 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:09 smithi083 ceph-mon[21788]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 1 op/s 2024-04-22T21:36:09.641 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:09 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-22T21:36:10.225 INFO:teuthology.orchestra.run.smithi083.stdout:NFS Cluster Created Successfully 2024-04-22T21:36:10.461 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:10 smithi083 ceph-mon[21788]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:10.461 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:10 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-22T21:36:10.461 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:10 smithi083 ceph-mon[21788]: osdmap e65: 8 total, 8 up, 8 in 2024-04-22T21:36:10.461 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:10 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-22T21:36:10.469 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:10 smithi172 ceph-mon[23940]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:10.469 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:10 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-22T21:36:10.469 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:10 smithi172 ceph-mon[23940]: osdmap e65: 8 total, 8 up, 8 in 2024-04-22T21:36:10.469 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:10 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-22T21:36:10.928 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-04-22T21:36:11.501 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-22T21:36:11.501 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: osdmap e66: 8 total, 8 up, 8 in 2024-04-22T21:36:11.501 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: Saving service nfs.foo spec with placement count:2 2024-04-22T21:36:11.501 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:11.501 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:11.501 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: Creating key for client.nfs.foo.0.0.smithi083.qbfocx 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi083.qbfocx", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi083.qbfocx", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: Rados config object exists: conf-nfs.foo 2024-04-22T21:36:11.502 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: Creating key for client.nfs.foo.0.0.smithi083.qbfocx-rgw 2024-04-22T21:36:11.503 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi083.qbfocx-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:36:11.503 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi083.qbfocx-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:36:11.503 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:11.503 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: Deploying daemon nfs.foo.0.0.smithi083.qbfocx on smithi083 2024-04-22T21:36:11.503 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:11 smithi083 ceph-mon[21788]: pgmap v224: 97 pgs: 2 creating+activating, 5 creating+peering, 23 unknown, 67 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:36:11.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-22T21:36:11.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: osdmap e66: 8 total, 8 up, 8 in 2024-04-22T21:36:11.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: Saving service nfs.foo spec with placement count:2 2024-04-22T21:36:11.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:11.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:11.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:11.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:11.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: Creating key for client.nfs.foo.0.0.smithi083.qbfocx 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi083.qbfocx", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi083.qbfocx", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: Rados config object exists: conf-nfs.foo 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: Creating key for client.nfs.foo.0.0.smithi083.qbfocx-rgw 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi083.qbfocx-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:36:11.596 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi083.qbfocx-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:36:11.597 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:11.597 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: Deploying daemon nfs.foo.0.0.smithi083.qbfocx on smithi083 2024-04-22T21:36:11.597 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:11 smithi172 ceph-mon[23940]: pgmap v224: 97 pgs: 2 creating+activating, 5 creating+peering, 23 unknown, 67 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:36:12.565 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:12 smithi083 ceph-mon[21788]: osdmap e67: 8 total, 8 up, 8 in 2024-04-22T21:36:12.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:12 smithi172 ceph-mon[23940]: osdmap e67: 8 total, 8 up, 8 in 2024-04-22T21:36:13.569 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:13 smithi083 ceph-mon[21788]: mgrmap e20: smithi083.ochghz(active, since 5m), standbys: smithi172.edmjti 2024-04-22T21:36:13.569 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:13 smithi083 ceph-mon[21788]: pgmap v226: 97 pgs: 2 creating+activating, 5 creating+peering, 14 unknown, 76 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-22T21:36:13.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:13 smithi172 ceph-mon[23940]: mgrmap e20: smithi083.ochghz(active, since 5m), standbys: smithi172.edmjti 2024-04-22T21:36:13.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:13 smithi172 ceph-mon[23940]: pgmap v226: 97 pgs: 2 creating+activating, 5 creating+peering, 14 unknown, 76 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-22T21:36:14.004 INFO:teuthology.orchestra.run.smithi083.stderr:Invalid command: Unexpected argument '--clusterid' 2024-04-22T21:36:14.005 INFO:teuthology.orchestra.run.smithi083.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-04-22T21:36:14.005 INFO:teuthology.orchestra.run.smithi083.stderr:Error EINVAL: invalid command 2024-04-22T21:36:14.344 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-04-22T21:36:14.344 INFO:teuthology.orchestra.run.smithi083.stdout: "bind": "/fake", 2024-04-22T21:36:14.344 INFO:teuthology.orchestra.run.smithi083.stdout: "fs": "foofs", 2024-04-22T21:36:14.344 INFO:teuthology.orchestra.run.smithi083.stdout: "path": "/", 2024-04-22T21:36:14.344 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster": "foo", 2024-04-22T21:36:14.344 INFO:teuthology.orchestra.run.smithi083.stdout: "mode": "RW" 2024-04-22T21:36:14.344 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-04-22T21:36:14.649 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:14 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-22T21:36:14.649 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:14 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-22T21:36:14.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:14 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-22T21:36:14.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:14 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-22T21:36:15.012 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-04-22T21:36:15.601 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:15 smithi083 ceph-mon[21788]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:15.601 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:15 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:15.601 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:15 smithi083 ceph-mon[21788]: Creating key for client.nfs.foo.1.0.smithi172.pzwela 2024-04-22T21:36:15.601 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:15 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi172.pzwela", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-22T21:36:15.601 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:15 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi172.pzwela", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-22T21:36:15.601 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:15 smithi083 ceph-mon[21788]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-22T21:36:15.601 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:15 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-22T21:36:15.602 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:15 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-22T21:36:15.602 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:15 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:15.602 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:15 smithi083 ceph-mon[21788]: pgmap v227: 97 pgs: 2 creating+activating, 5 creating+peering, 90 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-22T21:36:15.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:15 smithi172 ceph-mon[23940]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:15.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:15 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:15.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:15 smithi172 ceph-mon[23940]: Creating key for client.nfs.foo.1.0.smithi172.pzwela 2024-04-22T21:36:15.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:15 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi172.pzwela", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-22T21:36:15.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:15 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi172.pzwela", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-22T21:36:15.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:15 smithi172 ceph-mon[23940]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-22T21:36:15.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:15 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-22T21:36:15.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:15 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-22T21:36:15.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:15 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:15.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:15 smithi172 ceph-mon[23940]: pgmap v227: 97 pgs: 2 creating+activating, 5 creating+peering, 90 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-22T21:36:16.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:16 smithi083 ceph-mon[21788]: pgmap v228: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-22T21:36:17.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:16 smithi172 ceph-mon[23940]: pgmap v228: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-22T21:36:17.821 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:17 smithi172 ceph-mon[23940]: from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:17.821 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-22T21:36:17.821 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-22T21:36:17.821 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:17 smithi172 ceph-mon[23940]: Rados config object exists: conf-nfs.foo 2024-04-22T21:36:17.822 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:17 smithi172 ceph-mon[23940]: Creating key for client.nfs.foo.1.0.smithi172.pzwela-rgw 2024-04-22T21:36:17.822 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi172.pzwela-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:36:17.822 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi172.pzwela-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:36:17.822 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:17 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:17.822 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:17 smithi172 ceph-mon[23940]: Deploying daemon nfs.foo.1.0.smithi172.pzwela on smithi172 2024-04-22T21:36:17.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:17 smithi083 ceph-mon[21788]: from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:17.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-22T21:36:17.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-22T21:36:17.899 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:17 smithi083 ceph-mon[21788]: Rados config object exists: conf-nfs.foo 2024-04-22T21:36:17.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:17 smithi083 ceph-mon[21788]: Creating key for client.nfs.foo.1.0.smithi172.pzwela-rgw 2024-04-22T21:36:17.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi172.pzwela-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:36:17.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi172.pzwela-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:36:17.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:17 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:17.900 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:17 smithi083 ceph-mon[21788]: Deploying daemon nfs.foo.1.0.smithi172.pzwela on smithi172 2024-04-22T21:36:19.460 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:19 smithi172 ceph-mon[23940]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:19.460 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:19 smithi172 ceph-mon[23940]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-04-22T21:36:19.649 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:19 smithi083 ceph-mon[21788]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:19.649 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:19 smithi083 ceph-mon[21788]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-04-22T21:36:20.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:20 smithi172 ceph-mon[23940]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:20.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:20 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:20.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:20 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:20.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:20 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:20.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:20 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:20.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:20 smithi083 ceph-mon[21788]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:20.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:20 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:20.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:20 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:20.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:20 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:20.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:20 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:21.539 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:21 smithi083 ceph-mon[21788]: pgmap v230: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-04-22T21:36:21.539 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:21 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:21.561 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:21 smithi172 ceph-mon[23940]: pgmap v230: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-04-22T21:36:21.561 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:21 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:22.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:22 smithi083 ceph-mon[21788]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:22.524 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:22 smithi172 ceph-mon[23940]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:23.476 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:23 smithi083 ceph-mon[21788]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:23.476 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:23 smithi083 ceph-mon[21788]: pgmap v231: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-22T21:36:23.476 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:23 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:23.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:23 smithi172 ceph-mon[23940]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:23.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:23 smithi172 ceph-mon[23940]: pgmap v231: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-22T21:36:23.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:23 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:24.344 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:24 smithi172 ceph-mon[23940]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:24.494 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:24 smithi083 ceph-mon[21788]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:24.810 INFO:teuthology.orchestra.run.smithi083.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-04-22T21:36:25.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:25 smithi083 ceph-mon[21788]: pgmap v232: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-22T21:36:25.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:25 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:25.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:25 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:25.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:25 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:25.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:25 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:25.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:25 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:25.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:25 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:25.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:25 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:25.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:25 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:25.572 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-22T21:36:25.582 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi083.front.sepia.ceph.com 2024-04-22T21:36:25.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-04-22T21:36:25.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:25 smithi172 ceph-mon[23940]: pgmap v232: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-22T21:36:25.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:25 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:25.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:25 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:25.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:25 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:25.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:25 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:25.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:25 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:25.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:25 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:25.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:25 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:25.595 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:25 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:25.607 INFO:teuthology.orchestra.run.smithi083.stderr:+ mkdir /mnt/foo 2024-04-22T21:36:25.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-04-22T21:36:25.678 INFO:teuthology.orchestra.run.smithi083.stderr:++ hostname 2024-04-22T21:36:25.679 INFO:teuthology.orchestra.run.smithi083.stderr:+ mount -t nfs smithi083:/fake /mnt/foo -o sync 2024-04-22T21:36:26.089 DEBUG:teuthology.orchestra.run.smithi083:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-04-22T21:36:26.153 INFO:teuthology.orchestra.run.smithi083.stderr:+ echo test 2024-04-22T21:36:26.165 DEBUG:teuthology.orchestra.run.smithi083:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-04-22T21:36:26.228 INFO:teuthology.orchestra.run.smithi083.stderr:+ sync 2024-04-22T21:36:26.399 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:26 smithi083 ceph-mon[21788]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:26.494 INFO:teuthology.run_tasks:Running task parallel... 2024-04-22T21:36:26.505 INFO:teuthology.task.parallel:starting parallel... 2024-04-22T21:36:26.505 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T21:36:26.505 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T21:36:26.506 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi083.front.sepia.ceph.com 2024-04-22T21:36:26.506 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-22T21:36:26.506 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T21:36:26.506 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-22T21:36:26.507 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T21:36:26.507 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi083.front.sepia.ceph.com 2024-04-22T21:36:26.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-04-22T21:36:26.572 INFO:teuthology.orchestra.run.smithi083.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-22T21:36:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-04-22T21:36:26.572 INFO:teuthology.orchestra.run.smithi083.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-22T21:36:26.573 INFO:teuthology.orchestra.run.smithi083.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-22T21:36:26.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:26 smithi172 ceph-mon[23940]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:26.673 INFO:teuthology.orchestra.run.smithi083.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-22T21:36:26.673 INFO:teuthology.orchestra.run.smithi083.stdout:releasing clients 2024-04-22T21:36:27.254 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:27 smithi083 ceph-mon[21788]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-22T21:36:27.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:27 smithi172 ceph-mon[23940]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-22T21:36:27.674 INFO:teuthology.orchestra.run.smithi083.stdout: 5 115 26.27 MB/sec warmup 1 sec latency 53.901 ms 2024-04-22T21:36:28.674 INFO:teuthology.orchestra.run.smithi083.stdout: 5 271 25.33 MB/sec warmup 2 sec latency 25.411 ms 2024-04-22T21:36:29.397 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:29 smithi083 ceph-mon[21788]: pgmap v234: 97 pgs: 97 active+clean; 8.6 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 693 KiB/s wr, 16 op/s 2024-04-22T21:36:29.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:29 smithi172 ceph-mon[23940]: pgmap v234: 97 pgs: 97 active+clean; 8.6 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 693 KiB/s wr, 16 op/s 2024-04-22T21:36:29.673 INFO:teuthology.orchestra.run.smithi083.stdout: 5 422 24.80 MB/sec warmup 3 sec latency 30.433 ms 2024-04-22T21:36:29.999 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-22T21:36:30.674 INFO:teuthology.orchestra.run.smithi083.stdout: 5 575 24.46 MB/sec warmup 4 sec latency 39.434 ms 2024-04-22T21:36:30.703 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-04-22T21:36:30.703 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-04-22T21:36:30.704 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-22T21:36:30.704 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:30 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-04-22T21:36:30.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-04-22T21:36:30.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-04-22T21:36:30.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-22T21:36:30.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:30 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-04-22T21:36:31.674 INFO:teuthology.orchestra.run.smithi083.stdout: 5 936 21.88 MB/sec warmup 5 sec latency 22.976 ms 2024-04-22T21:36:31.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:31 smithi083 ceph-mon[21788]: pgmap v235: 97 pgs: 97 active+clean; 26 MiB data, 75 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.3 MiB/s wr, 167 op/s 2024-04-22T21:36:31.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-04-22T21:36:31.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-04-22T21:36:31.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-22T21:36:31.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-04-22T21:36:31.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:31 smithi083 ceph-mon[21788]: osdmap e68: 8 total, 8 up, 8 in 2024-04-22T21:36:31.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:36:31.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:36:31.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:31 smithi083 ceph-mon[21788]: osdmap e69: 8 total, 8 up, 8 in 2024-04-22T21:36:32.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:31 smithi172 ceph-mon[23940]: pgmap v235: 97 pgs: 97 active+clean; 26 MiB data, 75 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.3 MiB/s wr, 167 op/s 2024-04-22T21:36:32.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-04-22T21:36:32.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-04-22T21:36:32.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-22T21:36:32.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-04-22T21:36:32.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:31 smithi172 ceph-mon[23940]: osdmap e68: 8 total, 8 up, 8 in 2024-04-22T21:36:32.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:36:32.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:36:32.095 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:31 smithi172 ceph-mon[23940]: osdmap e69: 8 total, 8 up, 8 in 2024-04-22T21:36:32.674 INFO:teuthology.orchestra.run.smithi083.stdout: 5 1520 21.73 MB/sec warmup 6 sec latency 44.397 ms 2024-04-22T21:36:32.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:32 smithi083 ceph-mon[21788]: pgmap v238: 97 pgs: 3 peering, 94 active+clean; 83 MiB data, 143 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 11 MiB/s wr, 504 op/s 2024-04-22T21:36:33.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:32 smithi172 ceph-mon[23940]: pgmap v238: 97 pgs: 3 peering, 94 active+clean; 83 MiB data, 143 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 11 MiB/s wr, 504 op/s 2024-04-22T21:36:33.674 INFO:teuthology.orchestra.run.smithi083.stdout: 5 2450 20.16 MB/sec warmup 7 sec latency 20.469 ms 2024-04-22T21:36:34.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-22T21:36:34.675 INFO:teuthology.orchestra.run.smithi083.stdout: 5 3377 22.49 MB/sec warmup 8 sec latency 20.068 ms 2024-04-22T21:36:34.753 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:34 smithi083 ceph-mon[21788]: pgmap v239: 97 pgs: 3 peering, 94 active+clean; 106 MiB data, 154 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 14 MiB/s wr, 590 op/s 2024-04-22T21:36:35.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:34 smithi172 ceph-mon[23940]: pgmap v239: 97 pgs: 3 peering, 94 active+clean; 106 MiB data, 154 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 14 MiB/s wr, 590 op/s 2024-04-22T21:36:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: 5 3871 22.12 MB/sec warmup 9 sec latency 29.838 ms 2024-04-22T21:36:36.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 4476 20.44 MB/sec warmup 10 sec latency 39.925 ms 2024-04-22T21:36:36.784 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:36 smithi083 ceph-mon[21788]: pgmap v240: 97 pgs: 97 active+clean; 116 MiB data, 181 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 15 MiB/s wr, 724 op/s 2024-04-22T21:36:37.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:36 smithi172 ceph-mon[23940]: pgmap v240: 97 pgs: 97 active+clean; 116 MiB data, 181 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 15 MiB/s wr, 724 op/s 2024-04-22T21:36:37.675 INFO:teuthology.orchestra.run.smithi083.stdout: 5 5379 20.67 MB/sec warmup 11 sec latency 26.260 ms 2024-04-22T21:36:38.675 INFO:teuthology.orchestra.run.smithi083.stdout: 5 6008 19.75 MB/sec warmup 12 sec latency 51.965 ms 2024-04-22T21:36:39.256 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph mgr module enable nfs --force' 2024-04-22T21:36:39.296 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:39 smithi083 ceph-mon[21788]: pgmap v241: 97 pgs: 97 active+clean; 125 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 17 MiB/s wr, 888 op/s 2024-04-22T21:36:39.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:39 smithi172 ceph-mon[23940]: pgmap v241: 97 pgs: 97 active+clean; 125 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 17 MiB/s wr, 888 op/s 2024-04-22T21:36:39.675 INFO:teuthology.orchestra.run.smithi083.stdout: 5 6767 20.62 MB/sec warmup 13 sec latency 33.690 ms 2024-04-22T21:36:40.675 INFO:teuthology.orchestra.run.smithi083.stdout: 5 7348 21.02 MB/sec warmup 14 sec latency 40.566 ms 2024-04-22T21:36:41.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 7825 19.83 MB/sec warmup 15 sec latency 29.832 ms 2024-04-22T21:36:41.845 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:41 smithi083 ceph-mon[21788]: pgmap v242: 97 pgs: 97 active+clean; 128 MiB data, 280 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 14 MiB/s wr, 835 op/s 2024-04-22T21:36:42.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:41 smithi172 ceph-mon[23940]: pgmap v242: 97 pgs: 97 active+clean; 128 MiB data, 280 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 14 MiB/s wr, 835 op/s 2024-04-22T21:36:42.675 INFO:teuthology.orchestra.run.smithi083.stdout: 5 8284 19.31 MB/sec warmup 16 sec latency 23.245 ms 2024-04-22T21:36:43.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 9157 19.08 MB/sec warmup 17 sec latency 29.665 ms 2024-04-22T21:36:43.692 INFO:teuthology.orchestra.run.smithi083.stderr:module 'nfs' is already enabled 2024-04-22T21:36:43.859 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:43 smithi083 ceph-mon[21788]: pgmap v243: 97 pgs: 97 active+clean; 139 MiB data, 323 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 12 MiB/s wr, 879 op/s 2024-04-22T21:36:43.859 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:43 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/4151132130' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-22T21:36:44.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:43 smithi172 ceph-mon[23940]: pgmap v243: 97 pgs: 97 active+clean; 139 MiB data, 323 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 12 MiB/s wr, 879 op/s 2024-04-22T21:36:44.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:43 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/4151132130' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-22T21:36:44.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 9622 18.69 MB/sec warmup 18 sec latency 26.508 ms 2024-04-22T21:36:44.841 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-22T21:36:44.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:44 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/4151132130' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-22T21:36:44.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:44 smithi083 ceph-mon[21788]: mgrmap e21: smithi083.ochghz(active, since 6m), standbys: smithi172.edmjti 2024-04-22T21:36:45.093 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:44 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/4151132130' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-22T21:36:45.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:44 smithi172 ceph-mon[23940]: mgrmap e21: smithi083.ochghz(active, since 6m), standbys: smithi172.edmjti 2024-04-22T21:36:45.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 10223 19.04 MB/sec warmup 19 sec latency 61.076 ms 2024-04-22T21:36:45.718 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:45 smithi083 ceph-mon[21788]: pgmap v244: 97 pgs: 97 active+clean; 143 MiB data, 349 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 11 MiB/s wr, 816 op/s 2024-04-22T21:36:46.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:45 smithi172 ceph-mon[23940]: pgmap v244: 97 pgs: 97 active+clean; 143 MiB data, 349 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 11 MiB/s wr, 816 op/s 2024-04-22T21:36:46.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 10911 19.51 MB/sec warmup 20 sec latency 31.104 ms 2024-04-22T21:36:47.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 11437 18.75 MB/sec warmup 21 sec latency 23.518 ms 2024-04-22T21:36:47.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:47 smithi083 ceph-mon[21788]: pgmap v245: 97 pgs: 97 active+clean; 145 MiB data, 382 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 828 op/s 2024-04-22T21:36:48.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:47 smithi172 ceph-mon[23940]: pgmap v245: 97 pgs: 97 active+clean; 145 MiB data, 382 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 828 op/s 2024-04-22T21:36:48.422 INFO:teuthology.orchestra.run.smithi083.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:36:48.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 12010 18.55 MB/sec warmup 22 sec latency 48.599 ms 2024-04-22T21:36:49.410 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T21:36:49.410 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi083.front.sepia.ceph.com 2024-04-22T21:36:49.411 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-22T21:36:49.494 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:49 smithi083 ceph-mon[21788]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:49.494 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:49 smithi083 ceph-mon[21788]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:36:49.494 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:49 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:49.495 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:49 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:49.495 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:49 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:49.495 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:49 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:49.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 12784 18.35 MB/sec warmup 23 sec latency 19.093 ms 2024-04-22T21:36:49.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:49 smithi172 ceph-mon[23940]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:49.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:49 smithi172 ceph-mon[23940]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:36:49.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:49 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:49.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:49 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:49.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:49 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:49.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:49 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 13258 18.18 MB/sec warmup 24 sec latency 21.838 ms 2024-04-22T21:36:50.755 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:50 smithi083 ceph-mon[21788]: pgmap v246: 97 pgs: 97 active+clean; 155 MiB data, 419 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-04-22T21:36:50.755 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:50.756 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:50.756 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:50 smithi083 ceph-mon[21788]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:36:50.756 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:50.756 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:50.756 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:50.756 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:50.756 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:50.756 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:50.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:50 smithi172 ceph-mon[23940]: pgmap v246: 97 pgs: 97 active+clean; 155 MiB data, 419 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-04-22T21:36:50.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:50.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:50.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:50 smithi172 ceph-mon[23940]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:36:50.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:50.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:50.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:50.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:50.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:50.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:36:51.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 13798 18.33 MB/sec warmup 25 sec latency 25.274 ms 2024-04-22T21:36:51.779 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:51 smithi083 ceph-mon[21788]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T21:36:51.779 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:51 smithi083 ceph-mon[21788]: pgmap v247: 97 pgs: 97 active+clean; 154 MiB data, 447 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 822 op/s 2024-04-22T21:36:51.779 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:51 smithi083 ceph-mon[21788]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-22T21:36:51.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:51 smithi172 ceph-mon[23940]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T21:36:51.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:51 smithi172 ceph-mon[23940]: pgmap v247: 97 pgs: 97 active+clean; 154 MiB data, 447 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 822 op/s 2024-04-22T21:36:51.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:51 smithi172 ceph-mon[23940]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-22T21:36:52.375 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-04-22T21:36:52.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 14411 18.58 MB/sec warmup 26 sec latency 41.908 ms 2024-04-22T21:36:52.867 INFO:teuthology.orchestra.run.smithi083.stdout:"Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-22T21:36:53.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 14790 18.07 MB/sec warmup 27 sec latency 26.180 ms 2024-04-22T21:36:53.802 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:53 smithi083 ceph-mon[21788]: pgmap v248: 97 pgs: 97 active+clean; 158 MiB data, 480 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.9 MiB/s wr, 963 op/s 2024-04-22T21:36:53.802 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:53 smithi083 ceph-mon[21788]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:53.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:53 smithi172 ceph-mon[23940]: pgmap v248: 97 pgs: 97 active+clean; 158 MiB data, 480 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.9 MiB/s wr, 963 op/s 2024-04-22T21:36:53.845 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:53 smithi172 ceph-mon[23940]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:54.505 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch ps' 2024-04-22T21:36:54.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 15572 18.00 MB/sec warmup 28 sec latency 30.162 ms 2024-04-22T21:36:54.810 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:54 smithi083 ceph-mon[21788]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:54.843 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:54 smithi172 ceph-mon[23940]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:55.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 16402 17.87 MB/sec warmup 29 sec latency 31.402 ms 2024-04-22T21:36:55.819 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:55 smithi083 ceph-mon[21788]: pgmap v249: 97 pgs: 97 active+clean; 161 MiB data, 513 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 811 op/s 2024-04-22T21:36:55.843 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:55 smithi172 ceph-mon[23940]: pgmap v249: 97 pgs: 97 active+clean; 161 MiB data, 513 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 811 op/s 2024-04-22T21:36:56.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 16870 17.84 MB/sec warmup 30 sec latency 44.571 ms 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.smithi083 smithi083 *:9093,9094 running (5m) 32s ago 6m 23.2M - ba2b418f427c b40e5826cf4a 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:crash.smithi083 smithi083 running (6m) 32s ago 6m 6409k - 17.2.0 e1d6a67b021e 8017849c4e67 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:crash.smithi172 smithi172 running (6m) 34s ago 6m 6409k - 17.2.0 e1d6a67b021e 579757f6e520 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.smithi083 smithi083 *:3000 running (5m) 32s ago 6m 50.0M - 8.3.5 dad864ee21e9 e6c664d97098 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:mds.foofs.smithi083.nyheuj smithi083 running (63s) 32s ago 62s 13.4M - 17.2.0 e1d6a67b021e 916c4f8c7802 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:mds.foofs.smithi172.cavhbk smithi172 running (60s) 34s ago 60s 11.2M - 17.2.0 e1d6a67b021e 72f5a264bb49 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.smithi083.ochghz smithi083 *:9283 running (8m) 32s ago 8m 460M - 17.2.0 e1d6a67b021e e6931af0de17 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.smithi172.edmjti smithi172 *:8443,9283 running (6m) 34s ago 6m 412M - 17.2.0 e1d6a67b021e 35ff72a3ae30 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:mon.smithi083 smithi083 running (8m) 32s ago 8m 50.4M 2048M 17.2.0 e1d6a67b021e 25f27fe2e2e7 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:mon.smithi172 smithi172 running (6m) 34s ago 6m 36.7M 2048M 17.2.0 e1d6a67b021e 2c168ec547f8 2024-04-22T21:36:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:nfs.foo.0.0.smithi083.qbfocx smithi083 *:2049 running (43s) 32s ago 43s 49.0M - 4.0 e1d6a67b021e 90569b05a39a 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:nfs.foo.1.0.smithi172.pzwela smithi172 *:2049 running (38s) 34s ago 37s 48.8M - 4.0 e1d6a67b021e 1a1fcfd1152f 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.smithi083 smithi083 *:9100 running (6m) 32s ago 6m 18.7M - 1dbe0e931976 72d59242559d 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.smithi172 smithi172 *:9100 running (5m) 34s ago 5m 17.0M - 1dbe0e931976 111fb1992476 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (5m) 32s ago 5m 42.5M 3216M 17.2.0 e1d6a67b021e f33f10a14180 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (4m) 32s ago 4m 41.9M 3216M 17.2.0 e1d6a67b021e fe80eaef93c7 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (4m) 32s ago 4m 41.4M 3216M 17.2.0 e1d6a67b021e cb42fed21ebe 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (4m) 32s ago 4m 39.6M 3216M 17.2.0 e1d6a67b021e f5943d14bc30 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi172 running (3m) 34s ago 3m 40.0M 3984M 17.2.0 e1d6a67b021e 2627d45f1040 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi172 running (3m) 34s ago 3m 39.6M 3984M 17.2.0 e1d6a67b021e 6bd4a960e7d5 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi172 running (2m) 34s ago 2m 38.9M 3984M 17.2.0 e1d6a67b021e c5e86086662c 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi172 running (2m) 34s ago 2m 41.0M 3984M 17.2.0 e1d6a67b021e 80378e864f5e 2024-04-22T21:36:57.481 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.smithi083 smithi083 *:9095 running (5m) 32s ago 5m 60.7M - 514e6a882f6e d268cb8349fb 2024-04-22T21:36:57.603 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:57 smithi083 ceph-mon[21788]: pgmap v250: 97 pgs: 97 active+clean; 166 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 893 op/s 2024-04-22T21:36:57.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 17558 18.17 MB/sec warmup 31 sec latency 31.846 ms 2024-04-22T21:36:57.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:57 smithi172 ceph-mon[23940]: pgmap v250: 97 pgs: 97 active+clean; 166 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 893 op/s 2024-04-22T21:36:58.613 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:58 smithi083 ceph-mon[21788]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:58.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 18093 18.18 MB/sec warmup 32 sec latency 49.671 ms 2024-04-22T21:36:58.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:58 smithi172 ceph-mon[23940]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:59.623 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:36:59 smithi083 ceph-mon[21788]: pgmap v251: 97 pgs: 97 active+clean; 169 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 9.3 MiB/s wr, 968 op/s 2024-04-22T21:36:59.676 INFO:teuthology.orchestra.run.smithi083.stdout: 5 18684 17.83 MB/sec warmup 33 sec latency 32.464 ms 2024-04-22T21:36:59.712 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions' 2024-04-22T21:36:59.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:36:59 smithi172 ceph-mon[23940]: pgmap v251: 97 pgs: 97 active+clean; 169 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 9.3 MiB/s wr, 968 op/s 2024-04-22T21:37:00.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 19231 17.73 MB/sec warmup 34 sec latency 48.494 ms 2024-04-22T21:37:01.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:01 smithi083 ceph-mon[21788]: pgmap v252: 97 pgs: 97 active+clean; 172 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.9 MiB/s wr, 783 op/s 2024-04-22T21:37:01.651 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:01 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:37:01.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 20036 17.58 MB/sec warmup 35 sec latency 97.477 ms 2024-04-22T21:37:01.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:01 smithi172 ceph-mon[23940]: pgmap v252: 97 pgs: 97 active+clean; 172 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.9 MiB/s wr, 783 op/s 2024-04-22T21:37:01.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:01 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:37:02.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 20890 18.07 MB/sec warmup 36 sec latency 21.825 ms 2024-04-22T21:37:03.001 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-04-22T21:37:03.001 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-04-22T21:37:03.001 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T21:37:03.001 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-04-22T21:37:03.001 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": { 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-04-22T21:37:03.002 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-04-22T21:37:03.654 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:03 smithi083 ceph-mon[21788]: pgmap v253: 97 pgs: 97 active+clean; 176 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.2 MiB/s wr, 922 op/s 2024-04-22T21:37:03.654 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:03 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/313456450' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:37:03.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 21537 18.22 MB/sec warmup 37 sec latency 86.503 ms 2024-04-22T21:37:03.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:03 smithi172 ceph-mon[23940]: pgmap v253: 97 pgs: 97 active+clean; 176 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.2 MiB/s wr, 922 op/s 2024-04-22T21:37:03.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:03 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/313456450' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:37:04.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 22019 17.92 MB/sec warmup 38 sec latency 39.713 ms 2024-04-22T21:37:04.683 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-22T21:37:05.673 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:05 smithi083 ceph-mon[21788]: pgmap v254: 97 pgs: 97 active+clean; 176 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.4 MiB/s wr, 731 op/s 2024-04-22T21:37:05.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 22581 17.80 MB/sec warmup 39 sec latency 38.310 ms 2024-04-22T21:37:05.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:05 smithi172 ceph-mon[23940]: pgmap v254: 97 pgs: 97 active+clean; 176 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.4 MiB/s wr, 731 op/s 2024-04-22T21:37:06.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 23431 17.73 MB/sec warmup 40 sec latency 17.960 ms 2024-04-22T21:37:07.423 INFO:teuthology.orchestra.run.smithi083.stdout:wait for servicemap items w/ changing names to refresh 2024-04-22T21:37:07.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 23961 17.66 MB/sec warmup 41 sec latency 20.796 ms 2024-04-22T21:37:07.692 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:07 smithi083 ceph-mon[21788]: pgmap v255: 97 pgs: 97 active+clean; 186 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 8.7 MiB/s wr, 842 op/s 2024-04-22T21:37:07.844 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:07 smithi172 ceph-mon[23940]: pgmap v255: 97 pgs: 97 active+clean; 186 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 8.7 MiB/s wr, 842 op/s 2024-04-22T21:37:08.450 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'sleep 60' 2024-04-22T21:37:08.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 24698 17.96 MB/sec warmup 42 sec latency 26.085 ms 2024-04-22T21:37:09.452 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:09 smithi083 ceph-mon[21788]: pgmap v256: 97 pgs: 97 active+clean; 193 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.6 MiB/s wr, 873 op/s 2024-04-22T21:37:09.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:09 smithi172 ceph-mon[23940]: pgmap v256: 97 pgs: 97 active+clean; 193 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.6 MiB/s wr, 873 op/s 2024-04-22T21:37:09.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 25178 17.98 MB/sec warmup 43 sec latency 54.438 ms 2024-04-22T21:37:10.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 25753 17.69 MB/sec warmup 44 sec latency 26.277 ms 2024-04-22T21:37:11.464 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:11 smithi083 ceph-mon[21788]: pgmap v257: 97 pgs: 97 active+clean; 191 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 785 op/s 2024-04-22T21:37:11.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:11 smithi172 ceph-mon[23940]: pgmap v257: 97 pgs: 97 active+clean; 191 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 785 op/s 2024-04-22T21:37:11.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 26478 17.72 MB/sec warmup 45 sec latency 18.232 ms 2024-04-22T21:37:12.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 27346 17.69 MB/sec warmup 46 sec latency 21.408 ms 2024-04-22T21:37:13.481 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:13 smithi083 ceph-mon[21788]: pgmap v258: 97 pgs: 97 active+clean; 200 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 815 op/s 2024-04-22T21:37:13.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:13 smithi172 ceph-mon[23940]: pgmap v258: 97 pgs: 97 active+clean; 200 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 815 op/s 2024-04-22T21:37:13.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 28059 17.96 MB/sec warmup 47 sec latency 22.444 ms 2024-04-22T21:37:14.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 28812 18.11 MB/sec warmup 48 sec latency 19.483 ms 2024-04-22T21:37:15.503 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:15 smithi083 ceph-mon[21788]: pgmap v259: 97 pgs: 97 active+clean; 204 MiB data, 828 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 742 op/s 2024-04-22T21:37:15.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:15 smithi172 ceph-mon[23940]: pgmap v259: 97 pgs: 97 active+clean; 204 MiB data, 828 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 742 op/s 2024-04-22T21:37:15.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 29421 17.93 MB/sec warmup 49 sec latency 40.170 ms 2024-04-22T21:37:16.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 30016 17.83 MB/sec warmup 50 sec latency 48.630 ms 2024-04-22T21:37:17.534 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:17 smithi083 ceph-mon[21788]: pgmap v260: 97 pgs: 97 active+clean; 213 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 929 op/s 2024-04-22T21:37:17.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:17 smithi172 ceph-mon[23940]: pgmap v260: 97 pgs: 97 active+clean; 213 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 929 op/s 2024-04-22T21:37:17.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 30719 17.75 MB/sec warmup 51 sec latency 40.581 ms 2024-04-22T21:37:18.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 31359 17.84 MB/sec warmup 52 sec latency 31.520 ms 2024-04-22T21:37:19.302 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:19 smithi083 ceph-mon[21788]: pgmap v261: 97 pgs: 97 active+clean; 216 MiB data, 902 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.9 MiB/s wr, 866 op/s 2024-04-22T21:37:19.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:19 smithi172 ceph-mon[23940]: pgmap v261: 97 pgs: 97 active+clean; 216 MiB data, 902 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.9 MiB/s wr, 866 op/s 2024-04-22T21:37:19.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 32113 18.12 MB/sec warmup 53 sec latency 45.518 ms 2024-04-22T21:37:20.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 32570 17.91 MB/sec warmup 54 sec latency 28.114 ms 2024-04-22T21:37:21.489 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:21 smithi083 ceph-mon[21788]: pgmap v262: 97 pgs: 97 active+clean; 221 MiB data, 941 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 863 op/s 2024-04-22T21:37:21.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:21 smithi172 ceph-mon[23940]: pgmap v262: 97 pgs: 97 active+clean; 221 MiB data, 941 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 863 op/s 2024-04-22T21:37:21.677 INFO:teuthology.orchestra.run.smithi083.stdout: 5 33259 17.82 MB/sec warmup 55 sec latency 40.609 ms 2024-04-22T21:37:22.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 33975 17.78 MB/sec warmup 56 sec latency 55.346 ms 2024-04-22T21:37:23.405 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:23 smithi083 ceph-mon[21788]: pgmap v263: 97 pgs: 97 active+clean; 226 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-22T21:37:23.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:23 smithi172 ceph-mon[23940]: pgmap v263: 97 pgs: 97 active+clean; 226 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-22T21:37:23.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 34669 17.76 MB/sec warmup 57 sec latency 29.577 ms 2024-04-22T21:37:24.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 35510 18.07 MB/sec warmup 58 sec latency 30.931 ms 2024-04-22T21:37:25.530 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:25 smithi083 ceph-mon[21788]: pgmap v264: 97 pgs: 97 active+clean; 226 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 916 op/s 2024-04-22T21:37:25.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:25 smithi172 ceph-mon[23940]: pgmap v264: 97 pgs: 97 active+clean; 226 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 916 op/s 2024-04-22T21:37:25.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 35982 17.98 MB/sec warmup 59 sec latency 25.747 ms 2024-04-22T21:37:26.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 36458 17.77 MB/sec warmup 60 sec latency 49.282 ms 2024-04-22T21:37:27.547 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:27 smithi083 ceph-mon[21788]: pgmap v265: 97 pgs: 97 active+clean; 228 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.1 MiB/s wr, 933 op/s 2024-04-22T21:37:27.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:27 smithi172 ceph-mon[23940]: pgmap v265: 97 pgs: 97 active+clean; 228 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.1 MiB/s wr, 933 op/s 2024-04-22T21:37:27.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 37078 17.73 MB/sec warmup 61 sec latency 28.861 ms 2024-04-22T21:37:28.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 37972 17.75 MB/sec warmup 62 sec latency 22.730 ms 2024-04-22T21:37:29.307 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:29 smithi083 ceph-mon[21788]: pgmap v266: 97 pgs: 97 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 855 op/s 2024-04-22T21:37:29.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:29 smithi172 ceph-mon[23940]: pgmap v266: 97 pgs: 97 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 855 op/s 2024-04-22T21:37:29.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 38732 17.97 MB/sec warmup 63 sec latency 47.246 ms 2024-04-22T21:37:30.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 39423 18.04 MB/sec warmup 64 sec latency 18.695 ms 2024-04-22T21:37:31.318 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:31 smithi083 ceph-mon[21788]: pgmap v267: 97 pgs: 97 active+clean; 238 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 872 op/s 2024-04-22T21:37:31.318 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:37:31.318 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:37:31.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:31 smithi172 ceph-mon[23940]: pgmap v267: 97 pgs: 97 active+clean; 238 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 872 op/s 2024-04-22T21:37:31.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:37:31.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:37:31.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 39940 17.89 MB/sec warmup 65 sec latency 75.561 ms 2024-04-22T21:37:32.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 40611 17.84 MB/sec warmup 66 sec latency 21.355 ms 2024-04-22T21:37:33.333 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:33 smithi083 ceph-mon[21788]: pgmap v268: 97 pgs: 97 active+clean; 242 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 891 op/s 2024-04-22T21:37:33.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:33 smithi172 ceph-mon[23940]: pgmap v268: 97 pgs: 97 active+clean; 242 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 891 op/s 2024-04-22T21:37:33.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 41463 17.83 MB/sec warmup 67 sec latency 21.252 ms 2024-04-22T21:37:34.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 42250 18.03 MB/sec warmup 68 sec latency 25.653 ms 2024-04-22T21:37:35.364 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:35 smithi083 ceph-mon[21788]: pgmap v269: 97 pgs: 97 active+clean; 244 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 712 op/s 2024-04-22T21:37:35.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:35 smithi172 ceph-mon[23940]: pgmap v269: 97 pgs: 97 active+clean; 244 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 712 op/s 2024-04-22T21:37:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 42854 18.10 MB/sec warmup 69 sec latency 23.298 ms 2024-04-22T21:37:36.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 43286 17.93 MB/sec warmup 70 sec latency 24.815 ms 2024-04-22T21:37:37.343 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:37 smithi083 ceph-mon[21788]: pgmap v270: 97 pgs: 97 active+clean; 253 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.5 MiB/s wr, 976 op/s 2024-04-22T21:37:37.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:37 smithi172 ceph-mon[23940]: pgmap v270: 97 pgs: 97 active+clean; 253 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.5 MiB/s wr, 976 op/s 2024-04-22T21:37:37.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 43749 17.78 MB/sec warmup 71 sec latency 37.022 ms 2024-04-22T21:37:38.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 44147 17.69 MB/sec warmup 72 sec latency 38.510 ms 2024-04-22T21:37:39.366 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:39 smithi083 ceph-mon[21788]: pgmap v271: 97 pgs: 97 active+clean; 254 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 924 op/s 2024-04-22T21:37:39.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:39 smithi172 ceph-mon[23940]: pgmap v271: 97 pgs: 97 active+clean; 254 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 924 op/s 2024-04-22T21:37:39.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 44830 17.65 MB/sec warmup 73 sec latency 31.648 ms 2024-04-22T21:37:40.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 45496 17.70 MB/sec warmup 74 sec latency 41.109 ms 2024-04-22T21:37:41.392 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:41 smithi083 ceph-mon[21788]: pgmap v272: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 856 op/s 2024-04-22T21:37:41.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:41 smithi172 ceph-mon[23940]: pgmap v272: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 856 op/s 2024-04-22T21:37:41.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 46243 17.80 MB/sec warmup 75 sec latency 27.178 ms 2024-04-22T21:37:42.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 46670 17.75 MB/sec warmup 76 sec latency 29.583 ms 2024-04-22T21:37:43.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:43 smithi083 ceph-mon[21788]: pgmap v273: 97 pgs: 97 active+clean; 261 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.2 MiB/s wr, 841 op/s 2024-04-22T21:37:43.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:43 smithi172 ceph-mon[23940]: pgmap v273: 97 pgs: 97 active+clean; 261 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.2 MiB/s wr, 841 op/s 2024-04-22T21:37:43.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 47267 17.64 MB/sec warmup 77 sec latency 70.105 ms 2024-04-22T21:37:44.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 47975 17.65 MB/sec warmup 78 sec latency 20.951 ms 2024-04-22T21:37:45.448 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:45 smithi083 ceph-mon[21788]: pgmap v274: 97 pgs: 97 active+clean; 266 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 805 op/s 2024-04-22T21:37:45.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:45 smithi172 ceph-mon[23940]: pgmap v274: 97 pgs: 97 active+clean; 266 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 805 op/s 2024-04-22T21:37:45.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 48685 17.57 MB/sec warmup 79 sec latency 54.215 ms 2024-04-22T21:37:46.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 49121 17.59 MB/sec warmup 80 sec latency 34.860 ms 2024-04-22T21:37:47.487 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:47 smithi083 ceph-mon[21788]: pgmap v275: 97 pgs: 97 active+clean; 273 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.9 MiB/s wr, 957 op/s 2024-04-22T21:37:47.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:47 smithi172 ceph-mon[23940]: pgmap v275: 97 pgs: 97 active+clean; 273 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.9 MiB/s wr, 957 op/s 2024-04-22T21:37:47.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 49962 17.78 MB/sec warmup 81 sec latency 27.462 ms 2024-04-22T21:37:48.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 50617 17.68 MB/sec warmup 82 sec latency 18.586 ms 2024-04-22T21:37:49.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:49 smithi083 ceph-mon[21788]: pgmap v276: 97 pgs: 97 active+clean; 275 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 797 op/s 2024-04-22T21:37:49.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:49 smithi172 ceph-mon[23940]: pgmap v276: 97 pgs: 97 active+clean; 275 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 797 op/s 2024-04-22T21:37:49.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 51274 17.65 MB/sec warmup 83 sec latency 18.311 ms 2024-04-22T21:37:50.513 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:50.513 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:50.513 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:50 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:50.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:50.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:50.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:50 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:50.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 52094 17.65 MB/sec warmup 84 sec latency 32.434 ms 2024-04-22T21:37:51.520 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:51 smithi083 ceph-mon[21788]: pgmap v277: 97 pgs: 97 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 773 op/s 2024-04-22T21:37:51.520 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:37:51.520 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:51 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:37:51.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:51 smithi172 ceph-mon[23940]: pgmap v277: 97 pgs: 97 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 773 op/s 2024-04-22T21:37:51.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:37:51.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:51 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' 2024-04-22T21:37:51.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 52918 17.80 MB/sec warmup 85 sec latency 20.016 ms 2024-04-22T21:37:52.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 53666 17.90 MB/sec warmup 86 sec latency 19.655 ms 2024-04-22T21:37:53.530 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:53 smithi083 ceph-mon[21788]: pgmap v278: 97 pgs: 97 active+clean; 286 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 1.02k op/s 2024-04-22T21:37:53.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:53 smithi172 ceph-mon[23940]: pgmap v278: 97 pgs: 97 active+clean; 286 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 1.02k op/s 2024-04-22T21:37:53.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 54339 17.79 MB/sec warmup 87 sec latency 16.962 ms 2024-04-22T21:37:54.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 55106 17.82 MB/sec warmup 88 sec latency 21.600 ms 2024-04-22T21:37:55.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:55 smithi083 ceph-mon[21788]: pgmap v279: 97 pgs: 97 active+clean; 288 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.7 MiB/s wr, 924 op/s 2024-04-22T21:37:55.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:55 smithi172 ceph-mon[23940]: pgmap v279: 97 pgs: 97 active+clean; 288 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.7 MiB/s wr, 924 op/s 2024-04-22T21:37:55.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 55950 17.84 MB/sec warmup 89 sec latency 28.373 ms 2024-04-22T21:37:56.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 56670 17.98 MB/sec warmup 90 sec latency 18.362 ms 2024-04-22T21:37:57.448 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:57 smithi083 ceph-mon[21788]: pgmap v280: 97 pgs: 97 active+clean; 292 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.2 MiB/s wr, 910 op/s 2024-04-22T21:37:57.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:57 smithi172 ceph-mon[23940]: pgmap v280: 97 pgs: 97 active+clean; 292 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.2 MiB/s wr, 910 op/s 2024-04-22T21:37:57.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 57413 18.01 MB/sec warmup 91 sec latency 22.409 ms 2024-04-22T21:37:58.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 58218 17.97 MB/sec warmup 92 sec latency 14.644 ms 2024-04-22T21:37:59.461 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:37:59 smithi083 ceph-mon[21788]: pgmap v281: 97 pgs: 97 active+clean; 299 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 908 op/s 2024-04-22T21:37:59.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:37:59 smithi172 ceph-mon[23940]: pgmap v281: 97 pgs: 97 active+clean; 299 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 908 op/s 2024-04-22T21:37:59.678 INFO:teuthology.orchestra.run.smithi083.stdout: 5 59050 17.99 MB/sec warmup 93 sec latency 24.168 ms 2024-04-22T21:38:00.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 59827 18.05 MB/sec warmup 94 sec latency 37.838 ms 2024-04-22T21:38:01.390 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:01 smithi083 ceph-mon[21788]: pgmap v282: 97 pgs: 97 active+clean; 301 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 904 op/s 2024-04-22T21:38:01.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:01 smithi172 ceph-mon[23940]: pgmap v282: 97 pgs: 97 active+clean; 301 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 904 op/s 2024-04-22T21:38:01.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 60583 18.12 MB/sec warmup 95 sec latency 19.831 ms 2024-04-22T21:38:02.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 61097 18.08 MB/sec warmup 96 sec latency 39.389 ms 2024-04-22T21:38:03.482 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:03 smithi083 ceph-mon[21788]: pgmap v283: 97 pgs: 97 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 12 MiB/s wr, 1.11k op/s 2024-04-22T21:38:03.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:03 smithi172 ceph-mon[23940]: pgmap v283: 97 pgs: 97 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 12 MiB/s wr, 1.11k op/s 2024-04-22T21:38:03.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 61886 18.04 MB/sec warmup 97 sec latency 19.864 ms 2024-04-22T21:38:04.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 62708 18.06 MB/sec warmup 98 sec latency 45.521 ms 2024-04-22T21:38:05.495 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:05 smithi083 ceph-mon[21788]: pgmap v284: 97 pgs: 97 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 827 op/s 2024-04-22T21:38:05.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:05 smithi172 ceph-mon[23940]: pgmap v284: 97 pgs: 97 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 827 op/s 2024-04-22T21:38:05.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 63399 18.10 MB/sec warmup 99 sec latency 23.491 ms 2024-04-22T21:38:06.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 64036 18.16 MB/sec warmup 100 sec latency 32.317 ms 2024-04-22T21:38:07.519 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:07 smithi083 ceph-mon[21788]: pgmap v285: 97 pgs: 97 active+clean; 321 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-22T21:38:07.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:07 smithi172 ceph-mon[23940]: pgmap v285: 97 pgs: 97 active+clean; 321 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-22T21:38:07.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 64571 18.13 MB/sec warmup 101 sec latency 25.169 ms 2024-04-22T21:38:08.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 65194 18.05 MB/sec warmup 102 sec latency 29.637 ms 2024-04-22T21:38:09.289 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:09 smithi083 ceph-mon[21788]: pgmap v286: 97 pgs: 97 active+clean; 323 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-04-22T21:38:09.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:09 smithi172 ceph-mon[23940]: pgmap v286: 97 pgs: 97 active+clean; 323 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-04-22T21:38:09.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 65795 17.99 MB/sec warmup 103 sec latency 75.950 ms 2024-04-22T21:38:10.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 66487 18.02 MB/sec warmup 104 sec latency 23.430 ms 2024-04-22T21:38:11.463 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:11 smithi083 ceph-mon[21788]: pgmap v287: 97 pgs: 97 active+clean; 324 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 954 op/s 2024-04-22T21:38:11.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:11 smithi172 ceph-mon[23940]: pgmap v287: 97 pgs: 97 active+clean; 324 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 954 op/s 2024-04-22T21:38:11.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 67023 18.02 MB/sec warmup 105 sec latency 30.748 ms 2024-04-22T21:38:12.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 67598 18.03 MB/sec warmup 106 sec latency 41.636 ms 2024-04-22T21:38:13.291 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch ps' 2024-04-22T21:38:13.480 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:13 smithi083 ceph-mon[21788]: pgmap v288: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 1.04k op/s 2024-04-22T21:38:13.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:13 smithi172 ceph-mon[23940]: pgmap v288: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 1.04k op/s 2024-04-22T21:38:13.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 68061 17.98 MB/sec warmup 107 sec latency 28.800 ms 2024-04-22T21:38:14.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 68504 17.88 MB/sec warmup 108 sec latency 41.295 ms 2024-04-22T21:38:15.499 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:15 smithi083 ceph-mon[21788]: pgmap v289: 97 pgs: 97 active+clean; 327 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 840 op/s 2024-04-22T21:38:15.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:15 smithi172 ceph-mon[23940]: pgmap v289: 97 pgs: 97 active+clean; 327 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 840 op/s 2024-04-22T21:38:15.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 68991 17.81 MB/sec warmup 109 sec latency 29.480 ms 2024-04-22T21:38:16.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 69620 17.81 MB/sec warmup 110 sec latency 49.597 ms 2024-04-22T21:38:16.914 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.smithi083 smithi083 *:9093,9094 running (6m) 112s ago 8m 23.2M - ba2b418f427c b40e5826cf4a 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:crash.smithi083 smithi083 running (8m) 112s ago 8m 6409k - 17.2.0 e1d6a67b021e 8017849c4e67 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:crash.smithi172 smithi172 running (7m) 114s ago 7m 6409k - 17.2.0 e1d6a67b021e 579757f6e520 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.smithi083 smithi083 *:3000 running (6m) 112s ago 7m 50.0M - 8.3.5 dad864ee21e9 e6c664d97098 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:mds.foofs.smithi083.nyheuj smithi083 running (2m) 112s ago 2m 13.4M - 17.2.0 e1d6a67b021e 916c4f8c7802 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:mds.foofs.smithi172.cavhbk smithi172 running (2m) 114s ago 2m 11.2M - 17.2.0 e1d6a67b021e 72f5a264bb49 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.smithi083.ochghz smithi083 *:9283 running (9m) 112s ago 9m 460M - 17.2.0 e1d6a67b021e e6931af0de17 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.smithi172.edmjti smithi172 *:8443,9283 running (7m) 114s ago 7m 412M - 17.2.0 e1d6a67b021e 35ff72a3ae30 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:mon.smithi083 smithi083 running (9m) 112s ago 9m 50.4M 2048M 17.2.0 e1d6a67b021e 25f27fe2e2e7 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:mon.smithi172 smithi172 running (7m) 114s ago 7m 36.7M 2048M 17.2.0 e1d6a67b021e 2c168ec547f8 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:nfs.foo.0.0.smithi083.qbfocx smithi083 *:2049 running (2m) 112s ago 2m 49.0M - 4.0 e1d6a67b021e 90569b05a39a 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:nfs.foo.1.0.smithi172.pzwela smithi172 *:2049 running (117s) 114s ago 116s 48.8M - 4.0 e1d6a67b021e 1a1fcfd1152f 2024-04-22T21:38:16.915 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.smithi083 smithi083 *:9100 running (7m) 112s ago 7m 18.7M - 1dbe0e931976 72d59242559d 2024-04-22T21:38:16.916 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.smithi172 smithi172 *:9100 running (7m) 114s ago 7m 17.0M - 1dbe0e931976 111fb1992476 2024-04-22T21:38:16.916 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (6m) 112s ago 6m 42.5M 3216M 17.2.0 e1d6a67b021e f33f10a14180 2024-04-22T21:38:16.916 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (6m) 112s ago 6m 41.9M 3216M 17.2.0 e1d6a67b021e fe80eaef93c7 2024-04-22T21:38:16.916 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (5m) 112s ago 5m 41.4M 3216M 17.2.0 e1d6a67b021e cb42fed21ebe 2024-04-22T21:38:16.916 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (5m) 112s ago 5m 39.6M 3216M 17.2.0 e1d6a67b021e f5943d14bc30 2024-04-22T21:38:16.916 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi172 running (5m) 114s ago 5m 40.0M 3984M 17.2.0 e1d6a67b021e 2627d45f1040 2024-04-22T21:38:16.916 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi172 running (4m) 114s ago 4m 39.6M 3984M 17.2.0 e1d6a67b021e 6bd4a960e7d5 2024-04-22T21:38:16.916 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi172 running (4m) 114s ago 4m 38.9M 3984M 17.2.0 e1d6a67b021e c5e86086662c 2024-04-22T21:38:16.916 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi172 running (3m) 114s ago 3m 41.0M 3984M 17.2.0 e1d6a67b021e 80378e864f5e 2024-04-22T21:38:16.916 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.smithi083 smithi083 *:9095 running (7m) 112s ago 7m 60.7M - 514e6a882f6e d268cb8349fb 2024-04-22T21:38:17.536 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:17 smithi083 ceph-mon[21788]: pgmap v290: 97 pgs: 97 active+clean; 335 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.1 MiB/s wr, 1.04k op/s 2024-04-22T21:38:17.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:17 smithi172 ceph-mon[23940]: pgmap v290: 97 pgs: 97 active+clean; 335 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.1 MiB/s wr, 1.04k op/s 2024-04-22T21:38:17.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 70169 17.77 MB/sec warmup 111 sec latency 34.641 ms 2024-04-22T21:38:18.181 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch upgrade status' 2024-04-22T21:38:18.541 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:18 smithi083 ceph-mon[21788]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:18.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:18 smithi172 ceph-mon[23940]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:18.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 70656 17.76 MB/sec warmup 112 sec latency 31.586 ms 2024-04-22T21:38:19.547 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:19 smithi083 ceph-mon[21788]: pgmap v291: 97 pgs: 97 active+clean; 327 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.5 MiB/s wr, 778 op/s 2024-04-22T21:38:19.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:19 smithi172 ceph-mon[23940]: pgmap v291: 97 pgs: 97 active+clean; 327 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.5 MiB/s wr, 778 op/s 2024-04-22T21:38:19.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 71295 17.80 MB/sec warmup 113 sec latency 26.788 ms 2024-04-22T21:38:20.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 71886 17.76 MB/sec warmup 114 sec latency 41.084 ms 2024-04-22T21:38:21.568 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:21 smithi083 ceph-mon[21788]: pgmap v292: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.1 MiB/s wr, 829 op/s 2024-04-22T21:38:21.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:21 smithi172 ceph-mon[23940]: pgmap v292: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.1 MiB/s wr, 829 op/s 2024-04-22T21:38:21.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 72483 17.71 MB/sec warmup 115 sec latency 51.197 ms 2024-04-22T21:38:21.922 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-04-22T21:38:21.923 INFO:teuthology.orchestra.run.smithi083.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", 2024-04-22T21:38:21.923 INFO:teuthology.orchestra.run.smithi083.stdout: "in_progress": true, 2024-04-22T21:38:21.923 INFO:teuthology.orchestra.run.smithi083.stdout: "services_complete": [], 2024-04-22T21:38:21.923 INFO:teuthology.orchestra.run.smithi083.stdout: "progress": "", 2024-04-22T21:38:21.923 INFO:teuthology.orchestra.run.smithi083.stdout: "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-22T21:38:21.923 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-04-22T21:38:22.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 73206 17.72 MB/sec warmup 116 sec latency 27.603 ms 2024-04-22T21:38:22.887 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph health detail' 2024-04-22T21:38:23.340 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:23 smithi083 ceph-mon[21788]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:23.340 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:23 smithi083 ceph-mon[21788]: pgmap v293: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 897 op/s 2024-04-22T21:38:23.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:23 smithi172 ceph-mon[23940]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:23.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:23 smithi172 ceph-mon[23940]: pgmap v293: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 897 op/s 2024-04-22T21:38:23.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 73911 17.74 MB/sec warmup 117 sec latency 27.126 ms 2024-04-22T21:38:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 74430 17.77 MB/sec warmup 118 sec latency 49.430 ms 2024-04-22T21:38:25.363 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:25 smithi083 ceph-mon[21788]: pgmap v294: 97 pgs: 97 active+clean; 332 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 7.7 MiB/s wr, 709 op/s 2024-04-22T21:38:25.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:25 smithi172 ceph-mon[23940]: pgmap v294: 97 pgs: 97 active+clean; 332 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 7.7 MiB/s wr, 709 op/s 2024-04-22T21:38:25.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 74867 17.71 MB/sec warmup 119 sec latency 31.141 ms 2024-04-22T21:38:26.311 INFO:teuthology.orchestra.run.smithi083.stdout:HEALTH_WARN Upgrade: failed to pull target image 2024-04-22T21:38:26.311 INFO:teuthology.orchestra.run.smithi083.stdout:[WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T21:38:26.311 INFO:teuthology.orchestra.run.smithi083.stdout: host smithi083 `cephadm pull` failed: cephadm exited with an error code: 1, stderr: Pulling container image quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f... 2024-04-22T21:38:26.312 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 125 from /bin/podman pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:38:26.312 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f... 2024-04-22T21:38:26.312 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stderr Error: initializing source docker://quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f: reading manifest 430e09df97c8fc7dc2b2ae424f68ed11366c540f in quay.ceph.io/ceph-ci/ceph: manifest unknown 2024-04-22T21:38:26.312 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: Failed command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:38:26.371 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:26 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/3653193578' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T21:38:26.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:26 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/3653193578' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T21:38:27.294 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions' 2024-04-22T21:38:27.377 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:27 smithi083 ceph-mon[21788]: pgmap v295: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.0 MiB/s wr, 850 op/s 2024-04-22T21:38:27.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:27 smithi172 ceph-mon[23940]: pgmap v295: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.0 MiB/s wr, 850 op/s 2024-04-22T21:38:27.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 76050 11.80 MB/sec execute 1 sec latency 38.973 ms 2024-04-22T21:38:28.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 76686 14.37 MB/sec execute 2 sec latency 25.622 ms 2024-04-22T21:38:29.393 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:29 smithi083 ceph-mon[21788]: pgmap v296: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.5 MiB/s wr, 760 op/s 2024-04-22T21:38:29.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:29 smithi172 ceph-mon[23940]: pgmap v296: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.5 MiB/s wr, 760 op/s 2024-04-22T21:38:29.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 77294 14.54 MB/sec execute 3 sec latency 39.378 ms 2024-04-22T21:38:30.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 77922 17.11 MB/sec execute 4 sec latency 22.528 ms 2024-04-22T21:38:31.479 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:31 smithi083 ceph-mon[21788]: pgmap v297: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.5 MiB/s wr, 825 op/s 2024-04-22T21:38:31.479 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:38:31.479 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:31 smithi083 ceph-mon[21788]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:38:31.571 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": { 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": { 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": { 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": { 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: "overall": { 2024-04-22T21:38:31.572 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-22T21:38:31.573 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-04-22T21:38:31.573 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-04-22T21:38:31.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:31 smithi172 ceph-mon[23940]: pgmap v297: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.5 MiB/s wr, 825 op/s 2024-04-22T21:38:31.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/trash_purge_schedule"}]: dispatch 2024-04-22T21:38:31.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:31 smithi172 ceph-mon[23940]: from='mgr.14198 172.21.15.83:0/2570553526' entity='mgr.smithi083.ochghz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.ochghz/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:38:31.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 78388 16.49 MB/sec execute 5 sec latency 30.190 ms 2024-04-22T21:38:32.438 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-22T21:38:32.483 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:32 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/2640503556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:38:32.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:32 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/2640503556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:38:32.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 78881 15.88 MB/sec execute 6 sec latency 28.535 ms 2024-04-22T21:38:33.491 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:33 smithi083 ceph-mon[21788]: pgmap v298: 97 pgs: 97 active+clean; 325 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.2 MiB/s wr, 873 op/s 2024-04-22T21:38:33.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:33 smithi172 ceph-mon[23940]: pgmap v298: 97 pgs: 97 active+clean; 325 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.2 MiB/s wr, 873 op/s 2024-04-22T21:38:33.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 79481 14.84 MB/sec execute 7 sec latency 32.198 ms 2024-04-22T21:38:34.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 80093 15.36 MB/sec execute 8 sec latency 23.744 ms 2024-04-22T21:38:35.506 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:35 smithi083 ceph-mon[21788]: pgmap v299: 97 pgs: 97 active+clean; 324 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 816 op/s 2024-04-22T21:38:35.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:35 smithi172 ceph-mon[23940]: pgmap v299: 97 pgs: 97 active+clean; 324 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 816 op/s 2024-04-22T21:38:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 80721 15.21 MB/sec execute 9 sec latency 31.072 ms 2024-04-22T21:38:35.781 INFO:teuthology.orchestra.run.smithi083.stdout:true 2024-04-22T21:38:36.527 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:36 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/879127561' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:38:36.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:36 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/879127561' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:38:36.679 INFO:teuthology.orchestra.run.smithi083.stdout: 5 81195 15.32 MB/sec execute 10 sec latency 35.694 ms 2024-04-22T21:38:36.944 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-22T21:38:37.541 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:37 smithi083 ceph-mon[21788]: pgmap v300: 97 pgs: 97 active+clean; 316 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 8.4 MiB/s wr, 862 op/s 2024-04-22T21:38:37.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:37 smithi172 ceph-mon[23940]: pgmap v300: 97 pgs: 97 active+clean; 316 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 8.4 MiB/s wr, 862 op/s 2024-04-22T21:38:37.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 81767 15.61 MB/sec execute 11 sec latency 27.403 ms 2024-04-22T21:38:38.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 82215 15.50 MB/sec execute 12 sec latency 30.764 ms 2024-04-22T21:38:39.320 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:39 smithi083 ceph-mon[21788]: pgmap v301: 97 pgs: 97 active+clean; 313 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 793 op/s 2024-04-22T21:38:39.593 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:39 smithi172 ceph-mon[23940]: pgmap v301: 97 pgs: 97 active+clean; 313 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 793 op/s 2024-04-22T21:38:39.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 82830 15.09 MB/sec execute 13 sec latency 24.622 ms 2024-04-22T21:38:40.329 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:40 smithi083 ceph-mon[21788]: from='client.? 172.21.15.83:0/546492525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:38:40.594 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:40 smithi172 ceph-mon[23940]: from='client.? 172.21.15.83:0/546492525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:38:40.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 83392 14.86 MB/sec execute 14 sec latency 38.551 ms 2024-04-22T21:38:41.333 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:41 smithi083 ceph-mon[21788]: pgmap v302: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 738 op/s 2024-04-22T21:38:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:38:41.588 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-22T21:38:41.807 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f30207610fa44cdeb43ea797947eb85d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-22T21:38:41.811 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T21:38:41.823 INFO:tasks.cephadm:Teardown begin 2024-04-22T21:38:41.824 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:38:41.830 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:41 smithi172 ceph-mon[23940]: pgmap v302: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 738 op/s 2024-04-22T21:38:41.830 INFO:teuthology.orchestra.run.smithi083.stdout: 5 84042 15.15 MB/sec execute 15 sec latency 41.281 ms 2024-04-22T21:38:41.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:38:41.894 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T21:38:41.894 DEBUG:teuthology.orchestra.run.smithi083:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T21:38:41.919 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T21:38:41.945 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T21:38:41.945 INFO:tasks.cephadm.mon.smithi083:Stopping mon.smithi083... 2024-04-22T21:38:41.945 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mon.smithi083 2024-04-22T21:38:42.089 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:42 smithi083 systemd[1]: Stopping Ceph mon.smithi083 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:38:42.089 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:42 smithi083 bash[54549]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon.smithi083" found: no such container 2024-04-22T21:38:42.345 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:42 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083[21784]: 2024-04-22T21:38:42.252+0000 7f62e5a9c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi083 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:38:42.345 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:42 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083[21784]: 2024-04-22T21:38:42.252+0000 7f62e5a9c700 -1 mon.smithi083@0(leader) e2 *** Got Signal Terminated *** 2024-04-22T21:38:42.345 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:42 smithi083 podman[54560]: 2024-04-22 21:38:42.2952211 +0000 UTC m=+0.195848968 container died 25f27fe2e2e709a1032e4f2d5acc8c0372c682ba4d413cbae05a8ea22f7d4ca2 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, ceph=True, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , distribution-scope=public, architecture=x86_64, release=754, version=8, GIT_CLEAN=True, vcs-type=git, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container) 2024-04-22T21:38:42.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 84560 15.13 MB/sec execute 16 sec latency 37.066 ms 2024-04-22T21:38:43.104 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:42 smithi083 podman[54560]: 2024-04-22 21:38:42.872369774 +0000 UTC m=+0.772997648 container cleanup 25f27fe2e2e709a1032e4f2d5acc8c0372c682ba4d413cbae05a8ea22f7d4ca2 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.openshift.expose-services=, vcs-type=git, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, io.buildah.version=1.19.8, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, ceph=True, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , architecture=x86_64, vendor=Red Hat, Inc., distribution-scope=public) 2024-04-22T21:38:43.104 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:42 smithi083 bash[54560]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi083 2024-04-22T21:38:43.104 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Apr 22 21:38:42 smithi083 bash[54581]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon.smithi083" found: no such container 2024-04-22T21:38:43.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mon.smithi083.service' 2024-04-22T21:38:43.257 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:38:43.257 INFO:tasks.cephadm.mon.smithi083:Stopped mon.smithi083 2024-04-22T21:38:43.258 INFO:tasks.cephadm.mon.smithi172:Stopping mon.smithi172... 2024-04-22T21:38:43.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mon.smithi172 2024-04-22T21:38:43.536 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:43 smithi172 systemd[1]: Stopping Ceph mon.smithi172 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:38:43.536 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:43 smithi172 bash[40047]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon.smithi172" found: no such container 2024-04-22T21:38:43.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 15.59 MB/sec execute 17 sec latency 134.586 ms 2024-04-22T21:38:43.835 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:43 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi172[23933]: 2024-04-22T21:38:43.553+0000 7f16b11cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi172 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:38:43.836 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:43 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi172[23933]: 2024-04-22T21:38:43.553+0000 7f16b11cd700 -1 mon.smithi172@1(peon) e2 *** Got Signal Terminated *** 2024-04-22T21:38:43.836 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:43 smithi172 podman[40060]: 2024-04-22 21:38:43.584750266 +0000 UTC m=+0.227638105 container died 2c168ec547f8ecbd497c5a5be96ed752dee8546324f0fb2b71e29e50f614971b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi172, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, architecture=x86_64, release=754, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, RELEASE=HEAD, version=8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T21:38:44.093 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:44 smithi172 podman[40060]: 2024-04-22 21:38:44.037157381 +0000 UTC m=+0.680045211 container cleanup 2c168ec547f8ecbd497c5a5be96ed752dee8546324f0fb2b71e29e50f614971b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi172, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_CLEAN=True, release=754, distribution-scope=public, vendor=Red Hat, Inc., io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, ceph=True, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T21:38:44.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:44 smithi172 bash[40060]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon-smithi172 2024-04-22T21:38:44.094 INFO:journalctl@ceph.mon.smithi172.smithi172.stdout:Apr 22 21:38:44 smithi172 bash[40085]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-mon.smithi172" found: no such container 2024-04-22T21:38:44.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@mon.smithi172.service' 2024-04-22T21:38:44.347 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:38:44.347 INFO:tasks.cephadm.mon.smithi172:Stopped mon.smithi172 2024-04-22T21:38:44.348 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T21:38:44.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.0 2024-04-22T21:38:44.649 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:44 smithi083 systemd[1]: Stopping Ceph osd.0 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:38:44.649 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:44 smithi083 bash[54632]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-22T21:38:44.649 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:44 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0[32122]: 2024-04-22T21:38:44.561+0000 7fe7839a8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:38:44.650 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:44 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0[32122]: 2024-04-22T21:38:44.561+0000 7fe7839a8700 -1 osd.0 69 *** Got signal Terminated *** 2024-04-22T21:38:44.650 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:44 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0[32122]: 2024-04-22T21:38:44.561+0000 7fe7839a8700 -1 osd.0 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:38:44.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 14.72 MB/sec execute 18 sec latency 1134.629 ms 2024-04-22T21:38:45.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 13.95 MB/sec execute 19 sec latency 2134.667 ms 2024-04-22T21:38:46.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 13.25 MB/sec execute 20 sec latency 3134.698 ms 2024-04-22T21:38:47.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 12.62 MB/sec execute 21 sec latency 4134.748 ms 2024-04-22T21:38:48.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 12.05 MB/sec execute 22 sec latency 5134.803 ms 2024-04-22T21:38:49.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 11.52 MB/sec execute 23 sec latency 6134.855 ms 2024-04-22T21:38:50.290 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:50 smithi083 podman[54645]: 2024-04-22 21:38:50.009752584 +0000 UTC m=+5.543387884 container died f33f10a14180932c7b2b6e5732166eb4ddc720a4176f5ea3406f03aa7cbebae1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.component=centos-stream-container, ceph=True, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, architecture=x86_64, name=centos-stream, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, version=8, RELEASE=HEAD, distribution-scope=public) 2024-04-22T21:38:50.648 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:50 smithi083 podman[54645]: 2024-04-22 21:38:50.416058426 +0000 UTC m=+5.949693728 container cleanup f33f10a14180932c7b2b6e5732166eb4ddc720a4176f5ea3406f03aa7cbebae1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-type=git, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , release=754, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, com.redhat.component=centos-stream-container, architecture=x86_64) 2024-04-22T21:38:50.649 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:50 smithi083 bash[54645]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0 2024-04-22T21:38:50.649 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:50 smithi083 bash[54677]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-22T21:38:50.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 11.04 MB/sec execute 24 sec latency 7134.905 ms 2024-04-22T21:38:51.031 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:50 smithi083 podman[54664]: 2024-04-22 21:38:50.700918859 +0000 UTC m=+0.688943115 container remove f33f10a14180932c7b2b6e5732166eb4ddc720a4176f5ea3406f03aa7cbebae1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, ceph=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, distribution-scope=public, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, com.redhat.component=centos-stream-container) 2024-04-22T21:38:51.031 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:50 smithi083 podman[54738]: 2024-04-22 21:38:50.910099901 +0000 UTC m=+0.095508466 container create 383debe90239364ba62d95fdb87a3a630124deec4b350930c6b91a109444f2bd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0-deactivate, release=754, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, version=8, vendor=Red Hat, Inc., ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, name=centos-stream, io.buildah.version=1.19.8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T21:38:51.032 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:50 smithi083 podman[54738]: 2024-04-22 21:38:50.833534184 +0000 UTC m=+0.018942773 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:38:51.399 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:51 smithi083 podman[54738]: 2024-04-22 21:38:51.162682486 +0000 UTC m=+0.348091051 container init 383debe90239364ba62d95fdb87a3a630124deec4b350930c6b91a109444f2bd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, name=centos-stream, GIT_CLEAN=True, release=754, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, version=8, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64) 2024-04-22T21:38:51.399 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:51 smithi083 podman[54738]: 2024-04-22 21:38:51.166452145 +0000 UTC m=+0.351860723 container start 383debe90239364ba62d95fdb87a3a630124deec4b350930c6b91a109444f2bd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, RELEASE=HEAD, release=754, distribution-scope=public, GIT_BRANCH=HEAD, architecture=x86_64, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8) 2024-04-22T21:38:51.399 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:51 smithi083 podman[54738]: 2024-04-22 21:38:51.221000008 +0000 UTC m=+0.406408583 container attach 383debe90239364ba62d95fdb87a3a630124deec4b350930c6b91a109444f2bd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0-deactivate, release=754, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vendor=Red Hat, Inc., io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T21:38:51.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 10.60 MB/sec execute 25 sec latency 8134.929 ms 2024-04-22T21:38:51.814 INFO:journalctl@ceph.osd.0.smithi083.stdout:Apr 22 21:38:51 smithi083 podman[54738]: 2024-04-22 21:38:51.48626017 +0000 UTC m=+0.671668743 container died 383debe90239364ba62d95fdb87a3a630124deec4b350930c6b91a109444f2bd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-0-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.buildah.version=1.19.8, ceph=True, vcs-type=git, version=8, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754) 2024-04-22T21:38:52.123 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T21:38:52.159 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:38:52.159 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T21:38:52.160 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T21:38:52.160 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.1 2024-04-22T21:38:52.398 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:52 smithi083 systemd[1]: Stopping Ceph osd.1 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:38:52.399 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:52 smithi083 bash[54818]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-22T21:38:52.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 10.19 MB/sec execute 26 sec latency 9135.250 ms 2024-04-22T21:38:52.898 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:52 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1[36173]: 2024-04-22T21:38:52.415+0000 7fce91301700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:38:52.899 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:52 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1[36173]: 2024-04-22T21:38:52.415+0000 7fce91301700 -1 osd.1 69 *** Got signal Terminated *** 2024-04-22T21:38:52.899 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:52 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1[36173]: 2024-04-22T21:38:52.415+0000 7fce91301700 -1 osd.1 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:38:53.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 9.82 MB/sec execute 27 sec latency 10135.306 ms 2024-04-22T21:38:54.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 9.46 MB/sec execute 28 sec latency 11135.373 ms 2024-04-22T21:38:55.680 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 9.14 MB/sec execute 29 sec latency 12135.402 ms 2024-04-22T21:38:56.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 8.83 MB/sec execute 30 sec latency 13135.429 ms 2024-04-22T21:38:57.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 8.55 MB/sec execute 31 sec latency 14135.457 ms 2024-04-22T21:38:58.193 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:57 smithi083 podman[54829]: 2024-04-22 21:38:57.939426346 +0000 UTC m=+5.627465851 container died fe80eaef93c70895750ff5ecacadcf3b033c0b90968dd8dae2d3dd74f6054887 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1, com.redhat.component=centos-stream-container, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, release=754, ceph=True, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vcs-type=git, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-22T21:38:58.535 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:58 smithi083 podman[54829]: 2024-04-22 21:38:58.328025423 +0000 UTC m=+6.016064927 container cleanup fe80eaef93c70895750ff5ecacadcf3b033c0b90968dd8dae2d3dd74f6054887 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, ceph=True, release=754, vcs-type=git, distribution-scope=public, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., version=8) 2024-04-22T21:38:58.535 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:58 smithi083 bash[54829]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1 2024-04-22T21:38:58.535 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:58 smithi083 bash[54860]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-22T21:38:58.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 8.28 MB/sec execute 32 sec latency 15135.484 ms 2024-04-22T21:38:58.883 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:58 smithi083 podman[54848]: 2024-04-22 21:38:58.554270181 +0000 UTC m=+0.602267885 container remove fe80eaef93c70895750ff5ecacadcf3b033c0b90968dd8dae2d3dd74f6054887 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1, ceph=True, io.k8s.display-name=CentOS Stream 8, release=754, io.openshift.expose-services=, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, name=centos-stream, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container) 2024-04-22T21:38:58.883 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:58 smithi083 podman[54927]: 2024-04-22 21:38:58.661373412 +0000 UTC m=+0.018310755 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:38:58.884 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:58 smithi083 podman[54927]: 2024-04-22 21:38:58.763865101 +0000 UTC m=+0.120802445 container create 140b611d24d553a5242442d0f6154eba44ea104ef62031288aabe5f27564ecda (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , version=8, com.redhat.component=centos-stream-container, architecture=x86_64, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, RELEASE=HEAD, vcs-type=git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True) 2024-04-22T21:38:59.149 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:58 smithi083 podman[54927]: 2024-04-22 21:38:58.991301202 +0000 UTC m=+0.348238546 container init 140b611d24d553a5242442d0f6154eba44ea104ef62031288aabe5f27564ecda (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, distribution-scope=public, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., release=754, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, RELEASE=HEAD, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-22T21:38:59.149 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:58 smithi083 podman[54927]: 2024-04-22 21:38:58.996997251 +0000 UTC m=+0.353934596 container start 140b611d24d553a5242442d0f6154eba44ea104ef62031288aabe5f27564ecda (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1-deactivate, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, version=8, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.component=centos-stream-container, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-22T21:38:59.149 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:59 smithi083 podman[54927]: 2024-04-22 21:38:59.057921749 +0000 UTC m=+0.414859104 container attach 140b611d24d553a5242442d0f6154eba44ea104ef62031288aabe5f27564ecda (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, version=8, architecture=x86_64, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, RELEASE=HEAD, GIT_BRANCH=HEAD, name=centos-stream, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container) 2024-04-22T21:38:59.618 INFO:journalctl@ceph.osd.1.smithi083.stdout:Apr 22 21:38:59 smithi083 podman[54927]: 2024-04-22 21:38:59.332258778 +0000 UTC m=+0.689196122 container died 140b611d24d553a5242442d0f6154eba44ea104ef62031288aabe5f27564ecda (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-1-deactivate, name=centos-stream, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vcs-type=git, release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, ceph=True, RELEASE=HEAD, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, distribution-scope=public, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-22T21:38:59.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 8.03 MB/sec execute 33 sec latency 16135.514 ms 2024-04-22T21:38:59.905 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T21:38:59.941 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:38:59.941 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T21:38:59.941 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T21:38:59.941 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.2 2024-04-22T21:39:00.149 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:00 smithi083 systemd[1]: Stopping Ceph osd.2 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:39:00.149 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:00 smithi083 bash[55005]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-22T21:39:00.649 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:00 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2[39878]: 2024-04-22T21:39:00.208+0000 7ff842741700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:39:00.649 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:00 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2[39878]: 2024-04-22T21:39:00.208+0000 7ff842741700 -1 osd.2 69 *** Got signal Terminated *** 2024-04-22T21:39:00.649 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:00 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2[39878]: 2024-04-22T21:39:00.208+0000 7ff842741700 -1 osd.2 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:39:00.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 7.79 MB/sec execute 34 sec latency 17135.539 ms 2024-04-22T21:39:01.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 7.57 MB/sec execute 35 sec latency 18135.565 ms 2024-04-22T21:39:02.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 7.36 MB/sec execute 36 sec latency 19135.616 ms 2024-04-22T21:39:03.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 7.16 MB/sec execute 37 sec latency 20135.643 ms 2024-04-22T21:39:04.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 6.97 MB/sec execute 38 sec latency 21135.669 ms 2024-04-22T21:39:05.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 6.80 MB/sec execute 39 sec latency 22135.695 ms 2024-04-22T21:39:05.956 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:05 smithi083 podman[55016]: 2024-04-22 21:39:05.68449177 +0000 UTC m=+5.592836345 container died cb42fed21ebe26279ef78e95bc82e291b174bcebdb13b6e5fadbd2edc10c98c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.buildah.version=1.19.8, vcs-type=git, com.redhat.component=centos-stream-container, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, release=754, GIT_BRANCH=HEAD, io.openshift.expose-services=, distribution-scope=public) 2024-04-22T21:39:06.263 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:06 smithi083 podman[55016]: 2024-04-22 21:39:06.047436642 +0000 UTC m=+5.955781228 container cleanup cb42fed21ebe26279ef78e95bc82e291b174bcebdb13b6e5fadbd2edc10c98c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-type=git, version=8, name=centos-stream, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T21:39:06.263 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:06 smithi083 bash[55016]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2 2024-04-22T21:39:06.263 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:06 smithi083 bash[55048]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-22T21:39:06.530 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:06 smithi083 podman[55036]: 2024-04-22 21:39:06.282006732 +0000 UTC m=+0.592410001 container remove cb42fed21ebe26279ef78e95bc82e291b174bcebdb13b6e5fadbd2edc10c98c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2, RELEASE=HEAD, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, ceph=True, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.openshift.expose-services=, maintainer=Guillaume Abrioux , vcs-type=git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T21:39:06.531 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:06 smithi083 podman[55111]: 2024-04-22 21:39:06.419415128 +0000 UTC m=+0.020983824 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:39:06.531 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:06 smithi083 podman[55111]: 2024-04-22 21:39:06.549579712 +0000 UTC m=+0.151148408 container create bbf62ff8f989f6758d8c1346229cd15fd7c89b44973752bc171070caba64fad8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2-deactivate, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, distribution-scope=public, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.openshift.expose-services=, version=8, name=centos-stream, build-date=2022-05-03T08:36:31.336870) 2024-04-22T21:39:06.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 6.63 MB/sec execute 40 sec latency 23135.723 ms 2024-04-22T21:39:06.825 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:06 smithi083 podman[55111]: 2024-04-22 21:39:06.785406083 +0000 UTC m=+0.386974779 container init bbf62ff8f989f6758d8c1346229cd15fd7c89b44973752bc171070caba64fad8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.component=centos-stream-container, release=754, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux ) 2024-04-22T21:39:06.825 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:06 smithi083 podman[55111]: 2024-04-22 21:39:06.788694934 +0000 UTC m=+0.390263636 container start bbf62ff8f989f6758d8c1346229cd15fd7c89b44973752bc171070caba64fad8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, release=754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, ceph=True, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.openshift.tags=base centos centos-stream, distribution-scope=public, vendor=Red Hat, Inc., io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , RELEASE=HEAD) 2024-04-22T21:39:06.825 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:06 smithi083 podman[55111]: 2024-04-22 21:39:06.843697652 +0000 UTC m=+0.445266349 container attach bbf62ff8f989f6758d8c1346229cd15fd7c89b44973752bc171070caba64fad8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2-deactivate, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.buildah.version=1.19.8, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, version=8, vendor=Red Hat, Inc.) 2024-04-22T21:39:07.398 INFO:journalctl@ceph.osd.2.smithi083.stdout:Apr 22 21:39:07 smithi083 podman[55111]: 2024-04-22 21:39:07.123670785 +0000 UTC m=+0.725239482 container died bbf62ff8f989f6758d8c1346229cd15fd7c89b44973752bc171070caba64fad8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-2-deactivate, com.redhat.component=centos-stream-container, release=754, architecture=x86_64, vcs-type=git, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.expose-services=, RELEASE=HEAD, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T21:39:07.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 6.46 MB/sec execute 41 sec latency 24135.747 ms 2024-04-22T21:39:07.839 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T21:39:07.897 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:39:07.898 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T21:39:07.898 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T21:39:07.898 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.3 2024-04-22T21:39:08.059 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:08 smithi083 systemd[1]: Stopping Ceph osd.3 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:39:08.398 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:08 smithi083 bash[55190]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-22T21:39:08.399 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:08 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3[43141]: 2024-04-22T21:39:08.188+0000 7fe61b214700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:39:08.399 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:08 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3[43141]: 2024-04-22T21:39:08.188+0000 7fe61b214700 -1 osd.3 69 *** Got signal Terminated *** 2024-04-22T21:39:08.399 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:08 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3[43141]: 2024-04-22T21:39:08.188+0000 7fe61b214700 -1 osd.3 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:39:08.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 6.31 MB/sec execute 42 sec latency 25135.796 ms 2024-04-22T21:39:09.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 6.16 MB/sec execute 43 sec latency 26135.859 ms 2024-04-22T21:39:10.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 6.02 MB/sec execute 44 sec latency 27135.907 ms 2024-04-22T21:39:11.148 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:10 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3[43141]: 2024-04-22T21:39:10.828+0000 7fe615bab700 -1 osd.3 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:47.262672+0000 front 2024-04-22T21:38:47.262710+0000 (oldest deadline 2024-04-22T21:39:10.762843+0000) 2024-04-22T21:39:11.594 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:11 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:11.178+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:11.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 5.89 MB/sec execute 45 sec latency 28135.955 ms 2024-04-22T21:39:12.148 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:11 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3[43141]: 2024-04-22T21:39:11.855+0000 7fe615bab700 -1 osd.3 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:47.262672+0000 front 2024-04-22T21:38:47.262710+0000 (oldest deadline 2024-04-22T21:39:10.762843+0000) 2024-04-22T21:39:12.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:12 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:12.206+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:12.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 5.76 MB/sec execute 46 sec latency 29136.004 ms 2024-04-22T21:39:13.148 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:12 smithi083 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3[43141]: 2024-04-22T21:39:12.883+0000 7fe615bab700 -1 osd.3 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:47.262672+0000 front 2024-04-22T21:38:47.262710+0000 (oldest deadline 2024-04-22T21:39:10.762843+0000) 2024-04-22T21:39:13.190 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:12 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:12.905+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:13.593 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:13 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:13.247+0000 7f69da47c700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405095+0000 front 2024-04-22T21:38:49.405171+0000 (oldest deadline 2024-04-22T21:39:12.305301+0000) 2024-04-22T21:39:13.594 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:13 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:13.209+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:13.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 5.64 MB/sec execute 47 sec latency 30136.050 ms 2024-04-22T21:39:14.057 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:13 smithi083 podman[55201]: 2024-04-22 21:39:13.811844839 +0000 UTC m=+5.724954806 container died f5943d14bc309128f3f1c8a292fcc0b477c2d4aa2c654298c51dbb9ddcb88e46 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, name=centos-stream, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., version=8, RELEASE=HEAD, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True) 2024-04-22T21:39:14.202 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:13 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:13.891+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:14.339 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:14 smithi083 podman[55201]: 2024-04-22 21:39:14.158378299 +0000 UTC m=+6.071488258 container cleanup f5943d14bc309128f3f1c8a292fcc0b477c2d4aa2c654298c51dbb9ddcb88e46 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.buildah.version=1.19.8, name=centos-stream, GIT_CLEAN=True, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T21:39:14.339 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:14 smithi083 bash[55201]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3 2024-04-22T21:39:14.339 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:14 smithi083 podman[55218]: 2024-04-22 21:39:14.30957415 +0000 UTC m=+0.492337299 container remove f5943d14bc309128f3f1c8a292fcc0b477c2d4aa2c654298c51dbb9ddcb88e46 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , ceph=True, name=centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, version=8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, distribution-scope=public, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T21:39:14.339 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:14 smithi083 bash[55234]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-22T21:39:14.593 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:14 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:14.260+0000 7f69da47c700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405095+0000 front 2024-04-22T21:38:49.405171+0000 (oldest deadline 2024-04-22T21:39:12.305301+0000) 2024-04-22T21:39:14.594 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:14 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:14.466+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:14.594 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:14 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:14.220+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:14.599 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:14 smithi083 podman[55296]: 2024-04-22 21:39:14.507998333 +0000 UTC m=+0.021015497 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:39:14.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 5.52 MB/sec execute 48 sec latency 31136.105 ms 2024-04-22T21:39:14.869 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:14 smithi083 podman[55296]: 2024-04-22 21:39:14.61875994 +0000 UTC m=+0.131777093 container create d50eda6943fc0094a653dfef580bc2fc25be53d1f04938ab436c8acfda12a64e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3-deactivate, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, release=754) 2024-04-22T21:39:14.869 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:14 smithi083 podman[55296]: 2024-04-22 21:39:14.888088874 +0000 UTC m=+0.401106025 container init d50eda6943fc0094a653dfef580bc2fc25be53d1f04938ab436c8acfda12a64e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, release=754, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, architecture=x86_64, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, maintainer=Guillaume Abrioux ) 2024-04-22T21:39:15.148 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:14 smithi083 podman[55296]: 2024-04-22 21:39:14.893485769 +0000 UTC m=+0.406502923 container start d50eda6943fc0094a653dfef580bc2fc25be53d1f04938ab436c8acfda12a64e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.openshift.expose-services=, architecture=x86_64, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, ceph=True, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, release=754, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public) 2024-04-22T21:39:15.149 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:14 smithi083 podman[55296]: 2024-04-22 21:39:14.979833004 +0000 UTC m=+0.492850163 container attach d50eda6943fc0094a653dfef580bc2fc25be53d1f04938ab436c8acfda12a64e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., architecture=x86_64, ceph=True, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-type=git, com.redhat.component=centos-stream-container, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.openshift.tags=base centos centos-stream) 2024-04-22T21:39:15.193 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:14 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:14.889+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:15.470 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:15 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:15.269+0000 7f69da47c700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405095+0000 front 2024-04-22T21:38:49.405171+0000 (oldest deadline 2024-04-22T21:39:12.305301+0000) 2024-04-22T21:39:15.470 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:15 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:15.211+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:15.522 INFO:journalctl@ceph.osd.3.smithi083.stdout:Apr 22 21:39:15 smithi083 podman[55296]: 2024-04-22 21:39:15.252226106 +0000 UTC m=+0.765243249 container died d50eda6943fc0094a653dfef580bc2fc25be53d1f04938ab436c8acfda12a64e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-3-deactivate, distribution-scope=public, architecture=x86_64, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.buildah.version=1.19.8, name=centos-stream) 2024-04-22T21:39:15.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 5.41 MB/sec execute 49 sec latency 32136.158 ms 2024-04-22T21:39:15.843 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:15 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:15.488+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:15.855 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T21:39:15.891 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:39:15.891 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T21:39:15.892 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T21:39:15.892 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.4 2024-04-22T21:39:16.201 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:15 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:15.909+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:16.202 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:15 smithi172 systemd[1]: Stopping Ceph osd.4 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:39:16.202 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:15 smithi172 bash[40153]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-22T21:39:16.202 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:16 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:16.105+0000 7f69ddae1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:39:16.202 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:16 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:16.105+0000 7f69ddae1700 -1 osd.4 69 *** Got signal Terminated *** 2024-04-22T21:39:16.202 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:16 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:16.105+0000 7f69ddae1700 -1 osd.4 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:39:16.594 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:16 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:16.277+0000 7f69da47c700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405095+0000 front 2024-04-22T21:38:49.405171+0000 (oldest deadline 2024-04-22T21:39:12.305301+0000) 2024-04-22T21:39:16.594 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:16 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:16.460+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:16.595 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:16 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:16.220+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:16.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 5.30 MB/sec execute 50 sec latency 33136.183 ms 2024-04-22T21:39:17.219 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:16 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:16.897+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:17.593 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:17 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:17.443+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:17.594 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:17 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:17.249+0000 7f69da47c700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405095+0000 front 2024-04-22T21:38:49.405171+0000 (oldest deadline 2024-04-22T21:39:12.305301+0000) 2024-04-22T21:39:17.594 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:17 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:17.237+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:17.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 5.20 MB/sec execute 51 sec latency 34136.248 ms 2024-04-22T21:39:18.191 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:17 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:17.924+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:18.451 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:18 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:18.253+0000 7f69da47c700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405095+0000 front 2024-04-22T21:38:49.405171+0000 (oldest deadline 2024-04-22T21:39:12.305301+0000) 2024-04-22T21:39:18.451 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:18 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:18.210+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:18.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 5.10 MB/sec execute 52 sec latency 35136.274 ms 2024-04-22T21:39:18.843 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:18 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:18.469+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:19.185 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:18 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:18.912+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:19.439 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:19 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:19.220+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:19.440 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:19 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:19.203+0000 7f69da47c700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405095+0000 front 2024-04-22T21:38:49.405171+0000 (oldest deadline 2024-04-22T21:39:12.305301+0000) 2024-04-22T21:39:19.440 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:19 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:19.203+0000 7f69da47c700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:56.306502+0000 front 2024-04-22T21:38:56.306632+0000 (oldest deadline 2024-04-22T21:39:18.606676+0000) 2024-04-22T21:39:19.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 5.00 MB/sec execute 53 sec latency 36136.329 ms 2024-04-22T21:39:19.843 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:19 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:19.458+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:19.843 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:19 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:19.458+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:54.606443+0000 front 2024-04-22T21:38:54.606486+0000 (oldest deadline 2024-04-22T21:39:18.706650+0000) 2024-04-22T21:39:20.142 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:19 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:19.885+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:20.468 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:20 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:20.161+0000 7f69da47c700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405095+0000 front 2024-04-22T21:38:49.405171+0000 (oldest deadline 2024-04-22T21:39:12.305301+0000) 2024-04-22T21:39:20.468 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:20 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4[28186]: 2024-04-22T21:39:20.161+0000 7f69da47c700 -1 osd.4 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:56.306502+0000 front 2024-04-22T21:38:56.306632+0000 (oldest deadline 2024-04-22T21:39:18.606676+0000) 2024-04-22T21:39:20.469 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:20 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:20.175+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:20.469 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:20 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:20.175+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:20.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.91 MB/sec execute 54 sec latency 37136.387 ms 2024-04-22T21:39:20.831 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:20 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:20.489+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:20.831 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:20 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:20.489+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:54.606443+0000 front 2024-04-22T21:38:54.606486+0000 (oldest deadline 2024-04-22T21:39:18.706650+0000) 2024-04-22T21:39:21.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:20 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:20.849+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:21.549 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:21 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:21.439+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:21.549 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:21 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:21.439+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:54.606443+0000 front 2024-04-22T21:38:54.606486+0000 (oldest deadline 2024-04-22T21:39:18.706650+0000) 2024-04-22T21:39:21.550 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:21 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:21.218+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:21.550 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:21 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:21.218+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:21.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.82 MB/sec execute 55 sec latency 38136.413 ms 2024-04-22T21:39:21.829 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:21 smithi172 podman[40164]: 2024-04-22 21:39:21.568239338 +0000 UTC m=+5.576000835 container died 2627d45f1040beca4a091be1bed565d383210eca10a2af0f220fe5c5ba92394d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, distribution-scope=public, release=754, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, ceph=True, version=8, GIT_CLEAN=True, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-22T21:39:22.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:21 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:21.862+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:22.094 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:22 smithi172 podman[40164]: 2024-04-22 21:39:22.005180552 +0000 UTC m=+6.012942049 container cleanup 2627d45f1040beca4a091be1bed565d383210eca10a2af0f220fe5c5ba92394d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, version=8, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, ceph=True, vcs-type=git, release=754, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-22T21:39:22.094 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:22 smithi172 bash[40164]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4 2024-04-22T21:39:22.573 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:22 smithi172 podman[40183]: 2024-04-22 21:39:22.298760115 +0000 UTC m=+0.724966232 container remove 2627d45f1040beca4a091be1bed565d383210eca10a2af0f220fe5c5ba92394d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4, distribution-scope=public, com.redhat.component=centos-stream-container, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_BRANCH=HEAD) 2024-04-22T21:39:22.574 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:22 smithi172 bash[40195]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-22T21:39:22.574 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:22 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:22.413+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:22.575 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:22 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:22.413+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:54.606443+0000 front 2024-04-22T21:38:54.606486+0000 (oldest deadline 2024-04-22T21:39:18.706650+0000) 2024-04-22T21:39:22.575 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:22 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:22.174+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:22.576 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:22 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:22.174+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:22.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.73 MB/sec execute 56 sec latency 39136.436 ms 2024-04-22T21:39:22.844 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:22 smithi172 podman[40260]: 2024-04-22 21:39:22.49580684 +0000 UTC m=+0.019518780 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:39:22.844 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:22 smithi172 podman[40260]: 2024-04-22 21:39:22.607712597 +0000 UTC m=+0.131424507 container create 0c226721846793d177610398a60174411265faae38d82fb9b1fae0eede7608ae (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4-deactivate, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vcs-type=git, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.openshift.tags=base centos centos-stream, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754) 2024-04-22T21:39:22.844 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:22 smithi172 podman[40260]: 2024-04-22 21:39:22.827256624 +0000 UTC m=+0.350968541 container init 0c226721846793d177610398a60174411265faae38d82fb9b1fae0eede7608ae (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4-deactivate, io.openshift.tags=base centos centos-stream, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, RELEASE=HEAD, GIT_BRANCH=HEAD, version=8, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, architecture=x86_64) 2024-04-22T21:39:22.844 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:22 smithi172 podman[40260]: 2024-04-22 21:39:22.830749594 +0000 UTC m=+0.354461505 container start 0c226721846793d177610398a60174411265faae38d82fb9b1fae0eede7608ae (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4-deactivate, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.openshift.expose-services=, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD) 2024-04-22T21:39:23.096 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:22 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:22.886+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:23.096 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:22 smithi172 podman[40260]: 2024-04-22 21:39:22.926950441 +0000 UTC m=+0.450662349 container attach 0c226721846793d177610398a60174411265faae38d82fb9b1fae0eede7608ae (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4-deactivate, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, release=754, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, maintainer=Guillaume Abrioux , architecture=x86_64, distribution-scope=public, ceph=True, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, version=8, com.redhat.component=centos-stream-container) 2024-04-22T21:39:23.411 INFO:journalctl@ceph.osd.4.smithi172.stdout:Apr 22 21:39:23 smithi172 podman[40260]: 2024-04-22 21:39:23.115563661 +0000 UTC m=+0.639275581 container died 0c226721846793d177610398a60174411265faae38d82fb9b1fae0eede7608ae (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-4-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, name=centos-stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, ceph=True, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T21:39:23.411 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:23 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:23.194+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:23.411 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:23 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:23.194+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:23.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.65 MB/sec execute 57 sec latency 40136.486 ms 2024-04-22T21:39:23.754 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:23 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:23.461+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:23.754 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:23 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:23.461+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:54.606443+0000 front 2024-04-22T21:38:54.606486+0000 (oldest deadline 2024-04-22T21:39:18.706650+0000) 2024-04-22T21:39:23.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T21:39:23.809 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:39:23.810 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T21:39:23.810 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T21:39:23.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.5 2024-04-22T21:39:24.040 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:23 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:23.854+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:24.040 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:23 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:23.854+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:24.041 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:23 smithi172 systemd[1]: Stopping Ceph osd.5 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:39:24.041 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:23 smithi172 bash[40351]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-22T21:39:24.041 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:24 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:24.058+0000 7f57ebe0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:39:24.041 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:24 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:24.058+0000 7f57ebe0e700 -1 osd.5 69 *** Got signal Terminated *** 2024-04-22T21:39:24.041 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:24 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:24.058+0000 7f57ebe0e700 -1 osd.5 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:39:24.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:24 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:24.174+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:24.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:24 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:24.174+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.57 MB/sec execute 58 sec latency 41136.512 ms 2024-04-22T21:39:24.810 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:24 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:24.506+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:24.810 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:24 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:24.506+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:54.606443+0000 front 2024-04-22T21:38:54.606486+0000 (oldest deadline 2024-04-22T21:39:18.706650+0000) 2024-04-22T21:39:25.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:24 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:24.828+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:25.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:24 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:24.828+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:25.456 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:25 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:25.135+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:25.456 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:25 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:25.135+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:25.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.49 MB/sec execute 59 sec latency 42136.560 ms 2024-04-22T21:39:25.762 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:25 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:25.475+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:25.762 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:25 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:25.475+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:54.606443+0000 front 2024-04-22T21:38:54.606486+0000 (oldest deadline 2024-04-22T21:39:18.706650+0000) 2024-04-22T21:39:26.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:25 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:25.780+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:26.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:25 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:25.780+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:26.499 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:26 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:26.115+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:26.499 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:26 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:26.115+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:26.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.42 MB/sec execute 60 sec latency 43136.588 ms 2024-04-22T21:39:26.846 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:26 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:26.517+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:26.847 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:26 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:26.517+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:54.606443+0000 front 2024-04-22T21:38:54.606486+0000 (oldest deadline 2024-04-22T21:39:18.706650+0000) 2024-04-22T21:39:26.847 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:26 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:26.732+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:26.847 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:26 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:26.732+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:26.847 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:26 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:26.732+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:04.888681+0000 front 2024-04-22T21:39:04.888641+0000 (oldest deadline 2024-04-22T21:39:26.588716+0000) 2024-04-22T21:39:27.543 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:27 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:27.157+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:27.543 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:27 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:27.157+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:27.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.34 MB/sec execute 61 sec latency 44136.623 ms 2024-04-22T21:39:27.843 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:27 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:27.562+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:27.844 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:27 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:27.562+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:54.606443+0000 front 2024-04-22T21:38:54.606486+0000 (oldest deadline 2024-04-22T21:39:18.706650+0000) 2024-04-22T21:39:27.844 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:27 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:27.562+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:01.607455+0000 front 2024-04-22T21:39:01.607440+0000 (oldest deadline 2024-04-22T21:39:26.907546+0000) 2024-04-22T21:39:27.844 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:27 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:27.760+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:27.844 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:27 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:27.760+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:27.844 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:27 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:27.760+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:04.888681+0000 front 2024-04-22T21:39:04.888641+0000 (oldest deadline 2024-04-22T21:39:26.588716+0000) 2024-04-22T21:39:28.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:28 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:28.109+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:28.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:28 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:28.109+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:28.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.27 MB/sec execute 62 sec latency 45136.643 ms 2024-04-22T21:39:28.843 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:28 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:28.563+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:49.405636+0000 front 2024-04-22T21:38:49.405322+0000 (oldest deadline 2024-04-22T21:39:13.505541+0000) 2024-04-22T21:39:28.844 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:28 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:28.563+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:54.606443+0000 front 2024-04-22T21:38:54.606486+0000 (oldest deadline 2024-04-22T21:39:18.706650+0000) 2024-04-22T21:39:28.844 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:28 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5[31153]: 2024-04-22T21:39:28.563+0000 7f57e87a9700 -1 osd.5 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:01.607455+0000 front 2024-04-22T21:39:01.607440+0000 (oldest deadline 2024-04-22T21:39:26.907546+0000) 2024-04-22T21:39:28.844 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:28 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:28.715+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:28.844 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:28 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:28.715+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:28.844 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:28 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:28.715+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:04.888681+0000 front 2024-04-22T21:39:04.888641+0000 (oldest deadline 2024-04-22T21:39:26.588716+0000) 2024-04-22T21:39:29.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:29 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:29.062+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:29.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:29 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:29.062+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:29.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.21 MB/sec execute 63 sec latency 46136.691 ms 2024-04-22T21:39:29.707 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:29 smithi172 podman[40364]: 2024-04-22 21:39:29.466908639 +0000 UTC m=+5.525065759 container died 6bd4a960e7d5c01ea4817317a33d96abacfe70dbf23d545bd363186cafd38c83 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, release=754, GIT_BRANCH=HEAD, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, ceph=True, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, version=8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-22T21:39:29.708 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:29 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:29.700+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:29.708 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:29 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:29.700+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:29.708 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:29 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:29.700+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:04.888681+0000 front 2024-04-22T21:39:04.888641+0000 (oldest deadline 2024-04-22T21:39:26.588716+0000) 2024-04-22T21:39:30.047 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:29 smithi172 podman[40364]: 2024-04-22 21:39:29.891970726 +0000 UTC m=+5.950127842 container cleanup 6bd4a960e7d5c01ea4817317a33d96abacfe70dbf23d545bd363186cafd38c83 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, vcs-type=git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.openshift.expose-services=, name=centos-stream, GIT_CLEAN=True, architecture=x86_64, release=754) 2024-04-22T21:39:30.047 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:29 smithi172 bash[40364]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5 2024-04-22T21:39:30.047 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:29 smithi172 bash[40397]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-22T21:39:30.343 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:30 smithi172 podman[40384]: 2024-04-22 21:39:30.159989029 +0000 UTC m=+0.689048966 container remove 6bd4a960e7d5c01ea4817317a33d96abacfe70dbf23d545bd363186cafd38c83 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, GIT_BRANCH=HEAD, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, version=8, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, name=centos-stream, maintainer=Guillaume Abrioux ) 2024-04-22T21:39:30.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:30 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:30.065+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:30.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:30 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:30.065+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:30.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:30 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:30.065+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:30.636 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:30 smithi172 podman[40463]: 2024-04-22 21:39:30.289615568 +0000 UTC m=+0.020008192 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:39:30.637 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:30 smithi172 podman[40463]: 2024-04-22 21:39:30.435908891 +0000 UTC m=+0.166301545 container create 99e3a348dba1328ae41038c4e0cd7a91629ab16a788ce38ac2fd01e64f20bffe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5-deactivate, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.openshift.tags=base centos centos-stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.component=centos-stream-container, name=centos-stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8) 2024-04-22T21:39:30.637 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:30 smithi172 podman[40463]: 2024-04-22 21:39:30.655753892 +0000 UTC m=+0.386146506 container init 99e3a348dba1328ae41038c4e0cd7a91629ab16a788ce38ac2fd01e64f20bffe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5-deactivate, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.openshift.expose-services=, name=centos-stream, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, ceph=True, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, architecture=x86_64) 2024-04-22T21:39:30.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.14 MB/sec execute 64 sec latency 47136.742 ms 2024-04-22T21:39:30.932 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:30 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:30.709+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:30.933 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:30 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:30.709+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:30.933 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:30 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:30.709+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:04.888681+0000 front 2024-04-22T21:39:04.888641+0000 (oldest deadline 2024-04-22T21:39:26.588716+0000) 2024-04-22T21:39:30.933 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:30 smithi172 podman[40463]: 2024-04-22 21:39:30.659042022 +0000 UTC m=+0.389434633 container start 99e3a348dba1328ae41038c4e0cd7a91629ab16a788ce38ac2fd01e64f20bffe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5-deactivate, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.openshift.expose-services=, RELEASE=HEAD, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_CLEAN=True, release=754, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-type=git) 2024-04-22T21:39:30.933 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:30 smithi172 podman[40463]: 2024-04-22 21:39:30.729804089 +0000 UTC m=+0.460196701 container attach 99e3a348dba1328ae41038c4e0cd7a91629ab16a788ce38ac2fd01e64f20bffe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5-deactivate, io.openshift.expose-services=, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, maintainer=Guillaume Abrioux , GIT_CLEAN=True, version=8, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, distribution-scope=public) 2024-04-22T21:39:31.230 INFO:journalctl@ceph.osd.5.smithi172.stdout:Apr 22 21:39:30 smithi172 podman[40463]: 2024-04-22 21:39:30.951970978 +0000 UTC m=+0.682363588 container died 99e3a348dba1328ae41038c4e0cd7a91629ab16a788ce38ac2fd01e64f20bffe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-5-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, name=centos-stream, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, release=754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, architecture=x86_64) 2024-04-22T21:39:31.231 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:31 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:31.047+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:31.231 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:31 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:31.047+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:31.231 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:31 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:31.047+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:31.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T21:39:31.667 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:39:31.667 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T21:39:31.668 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-22T21:39:31.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.6 2024-04-22T21:39:31.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.08 MB/sec execute 65 sec latency 48136.765 ms 2024-04-22T21:39:31.884 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:31 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:31.752+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:31.884 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:31 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:31.752+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:31.885 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:31 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:31.752+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:04.888681+0000 front 2024-04-22T21:39:04.888641+0000 (oldest deadline 2024-04-22T21:39:26.588716+0000) 2024-04-22T21:39:31.885 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:31 smithi172 systemd[1]: Stopping Ceph osd.6 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:39:31.885 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:31 smithi172 bash[40541]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-22T21:39:31.885 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:31 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:31.901+0000 7fdfb77e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:39:31.885 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:31 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:31.901+0000 7fdfb77e1700 -1 osd.6 69 *** Got signal Terminated *** 2024-04-22T21:39:31.885 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:31 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:31.901+0000 7fdfb77e1700 -1 osd.6 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:39:32.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:32 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:32.077+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:32.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:32 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:32.077+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:32.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:32 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:32.077+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:32.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 4.02 MB/sec execute 66 sec latency 49136.824 ms 2024-04-22T21:39:33.092 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:32 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:32.708+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:33.092 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:32 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:32.708+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:33.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:32 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:32.708+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:04.888681+0000 front 2024-04-22T21:39:04.888641+0000 (oldest deadline 2024-04-22T21:39:26.588716+0000) 2024-04-22T21:39:33.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:33 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:33.109+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:33.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:33 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:33.109+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:33.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:33 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:33.109+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:33.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.96 MB/sec execute 67 sec latency 50136.876 ms 2024-04-22T21:39:34.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:33 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:33.708+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:34.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:33 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:33.708+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:34.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:33 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:33.708+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:04.888681+0000 front 2024-04-22T21:39:04.888641+0000 (oldest deadline 2024-04-22T21:39:26.588716+0000) 2024-04-22T21:39:34.593 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:34 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:34.137+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:34.593 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:34 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:34.137+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:34.593 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:34 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:34.137+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:34.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.90 MB/sec execute 68 sec latency 51136.927 ms 2024-04-22T21:39:35.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:34 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:34.731+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:35.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:34 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:34.731+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:35.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:34 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:34.731+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:04.888681+0000 front 2024-04-22T21:39:04.888641+0000 (oldest deadline 2024-04-22T21:39:26.588716+0000) 2024-04-22T21:39:35.593 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:35 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:35.147+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:35.593 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:35 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:35.147+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:35.594 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:35 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:35.147+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:35.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.84 MB/sec execute 69 sec latency 52136.987 ms 2024-04-22T21:39:36.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:35 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:35.711+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:36.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:35 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:35.711+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:36.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:35 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:35.711+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:04.888681+0000 front 2024-04-22T21:39:04.888641+0000 (oldest deadline 2024-04-22T21:39:26.588716+0000) 2024-04-22T21:39:36.593 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:36 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:36.141+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:36.593 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:36 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:36.141+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:36.593 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:36 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:36.141+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:36.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.79 MB/sec execute 70 sec latency 53137.037 ms 2024-04-22T21:39:37.093 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:36 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:36.728+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:48.486687+0000 front 2024-04-22T21:38:48.486518+0000 (oldest deadline 2024-04-22T21:39:11.986558+0000) 2024-04-22T21:39:37.094 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:36 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:36.728+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:51.987095+0000 front 2024-04-22T21:38:51.986989+0000 (oldest deadline 2024-04-22T21:39:23.187643+0000) 2024-04-22T21:39:37.094 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:36 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:36.728+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:04.888681+0000 front 2024-04-22T21:39:04.888641+0000 (oldest deadline 2024-04-22T21:39:26.588716+0000) 2024-04-22T21:39:37.094 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:36 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6[34134]: 2024-04-22T21:39:36.728+0000 7fdfb2979700 -1 osd.6 69 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-04-22T21:39:11.895314+0000 front 2024-04-22T21:39:11.895358+0000 (oldest deadline 2024-04-22T21:39:35.989714+0000) 2024-04-22T21:39:37.509 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:37 smithi172 podman[40555]: 2024-04-22 21:39:37.291891353 +0000 UTC m=+5.487077428 container died c5e86086662c47bf1015e370960d883146d72ab80c7ae7a74e76b0d61f69d922 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, vcs-type=git, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.buildah.version=1.19.8, ceph=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, distribution-scope=public) 2024-04-22T21:39:37.509 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:37 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:37.141+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:37.509 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:37 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:37.141+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:37.510 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:37 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:37.141+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:37.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.73 MB/sec execute 71 sec latency 54137.084 ms 2024-04-22T21:39:37.843 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:37 smithi172 podman[40555]: 2024-04-22 21:39:37.685809013 +0000 UTC m=+5.880995129 container cleanup c5e86086662c47bf1015e370960d883146d72ab80c7ae7a74e76b0d61f69d922 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_CLEAN=True, release=754, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, RELEASE=HEAD, version=8, distribution-scope=public, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, io.openshift.tags=base centos centos-stream, vcs-type=git, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T21:39:37.843 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:37 smithi172 bash[40555]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6 2024-04-22T21:39:37.844 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:37 smithi172 bash[40587]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-22T21:39:38.244 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:38 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:38.132+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:38.244 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:38 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:38.132+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:38.244 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:38 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:38.132+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:38.244 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:38 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:38.132+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-04-22T21:39:12.665367+0000 front 2024-04-22T21:39:12.665384+0000 (oldest deadline 2024-04-22T21:39:37.866187+0000) 2024-04-22T21:39:38.244 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:37 smithi172 podman[40574]: 2024-04-22 21:39:37.903640162 +0000 UTC m=+0.606873707 container remove c5e86086662c47bf1015e370960d883146d72ab80c7ae7a74e76b0d61f69d922 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, architecture=x86_64, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.openshift.expose-services=) 2024-04-22T21:39:38.245 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:38 smithi172 podman[40652]: 2024-04-22 21:39:38.027588423 +0000 UTC m=+0.019219560 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:39:38.245 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:38 smithi172 podman[40652]: 2024-04-22 21:39:38.129772119 +0000 UTC m=+0.121403246 container create e8de20d7dd1e3df26740849800a6a85316c60c25ffb75bb968bf9d78b253c81c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6-deactivate, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, RELEASE=HEAD, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.buildah.version=1.19.8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.openshift.expose-services=, architecture=x86_64) 2024-04-22T21:39:38.593 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:38 smithi172 podman[40652]: 2024-04-22 21:39:38.373713841 +0000 UTC m=+0.365344969 container init e8de20d7dd1e3df26740849800a6a85316c60c25ffb75bb968bf9d78b253c81c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6-deactivate, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_CLEAN=True, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, GIT_BRANCH=HEAD) 2024-04-22T21:39:38.593 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:38 smithi172 podman[40652]: 2024-04-22 21:39:38.377044549 +0000 UTC m=+0.368675680 container start e8de20d7dd1e3df26740849800a6a85316c60c25ffb75bb968bf9d78b253c81c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, RELEASE=HEAD, release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., version=8, vcs-type=git, distribution-scope=public, io.openshift.expose-services=) 2024-04-22T21:39:38.593 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:38 smithi172 podman[40652]: 2024-04-22 21:39:38.473702826 +0000 UTC m=+0.465333972 container attach e8de20d7dd1e3df26740849800a6a85316c60c25ffb75bb968bf9d78b253c81c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.openshift.expose-services=, RELEASE=HEAD, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, vcs-type=git, architecture=x86_64, distribution-scope=public, maintainer=Guillaume Abrioux ) 2024-04-22T21:39:38.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.68 MB/sec execute 72 sec latency 55137.129 ms 2024-04-22T21:39:38.978 INFO:journalctl@ceph.osd.6.smithi172.stdout:Apr 22 21:39:38 smithi172 podman[40652]: 2024-04-22 21:39:38.662723183 +0000 UTC m=+0.654354318 container died e8de20d7dd1e3df26740849800a6a85316c60c25ffb75bb968bf9d78b253c81c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-6-deactivate, vcs-type=git, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, version=8, release=754, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, name=centos-stream, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T21:39:39.266 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:39 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:39.093+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:39.266 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:39 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:39.093+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:39.266 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:39 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:39.093+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:39.267 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:39 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:39.093+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-04-22T21:39:12.665367+0000 front 2024-04-22T21:39:12.665384+0000 (oldest deadline 2024-04-22T21:39:37.866187+0000) 2024-04-22T21:39:39.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.6.service' 2024-04-22T21:39:39.323 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:39:39.323 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-22T21:39:39.324 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-22T21:39:39.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.7 2024-04-22T21:39:39.553 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:39 smithi172 systemd[1]: Stopping Ceph osd.7 for 3a5a1c14-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:39:39.553 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:39 smithi172 bash[40730]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-22T21:39:39.553 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:39 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:39.570+0000 7f8ee9fc8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:39:39.553 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:39 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:39.570+0000 7f8ee9fc8700 -1 osd.7 69 *** Got signal Terminated *** 2024-04-22T21:39:39.554 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:39 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:39.570+0000 7f8ee9fc8700 -1 osd.7 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:39:39.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.63 MB/sec execute 73 sec latency 56137.153 ms 2024-04-22T21:39:40.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:40 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:40.063+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:40.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:40 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:40.063+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:40.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:40 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:40.064+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:40.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:40 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:40.064+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-04-22T21:39:12.665367+0000 front 2024-04-22T21:39:12.665384+0000 (oldest deadline 2024-04-22T21:39:37.866187+0000) 2024-04-22T21:39:40.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.58 MB/sec execute 74 sec latency 57137.176 ms 2024-04-22T21:39:41.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:41 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:41.028+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:41.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:41 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:41.028+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:41.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:41 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:41.028+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:41.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:41 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:41.028+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-04-22T21:39:12.665367+0000 front 2024-04-22T21:39:12.665384+0000 (oldest deadline 2024-04-22T21:39:37.866187+0000) 2024-04-22T21:39:41.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.53 MB/sec execute 75 sec latency 58137.226 ms 2024-04-22T21:39:42.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:42 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:42.065+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:42.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:42 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:42.065+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:42.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:42 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:42.065+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:42.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:42 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:42.065+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-04-22T21:39:12.665367+0000 front 2024-04-22T21:39:12.665384+0000 (oldest deadline 2024-04-22T21:39:37.866187+0000) 2024-04-22T21:39:42.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:42 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:42.065+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.172:6804 osd.4 since back 2024-04-22T21:39:17.866633+0000 front 2024-04-22T21:39:17.866659+0000 (oldest deadline 2024-04-22T21:39:41.966720+0000) 2024-04-22T21:39:42.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.49 MB/sec execute 76 sec latency 59137.271 ms 2024-04-22T21:39:43.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:43 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:43.018+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:43.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:43 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:43.018+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:43.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:43 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:43.018+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:43.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:43 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:43.018+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-04-22T21:39:12.665367+0000 front 2024-04-22T21:39:12.665384+0000 (oldest deadline 2024-04-22T21:39:37.866187+0000) 2024-04-22T21:39:43.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:43 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:43.018+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.172:6804 osd.4 since back 2024-04-22T21:39:17.866633+0000 front 2024-04-22T21:39:17.866659+0000 (oldest deadline 2024-04-22T21:39:41.966720+0000) 2024-04-22T21:39:43.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.44 MB/sec execute 77 sec latency 60137.307 ms 2024-04-22T21:39:44.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:44 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:44.003+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-04-22T21:38:45.861338+0000 front 2024-04-22T21:38:45.861285+0000 (oldest deadline 2024-04-22T21:39:10.561346+0000) 2024-04-22T21:39:44.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:44 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:44.003+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-04-22T21:38:55.762706+0000 front 2024-04-22T21:38:55.762724+0000 (oldest deadline 2024-04-22T21:39:19.763358+0000) 2024-04-22T21:39:44.343 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:44 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:44.003+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-04-22T21:39:03.264189+0000 front 2024-04-22T21:39:03.264299+0000 (oldest deadline 2024-04-22T21:39:29.164348+0000) 2024-04-22T21:39:44.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:44 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:44.003+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-04-22T21:39:12.665367+0000 front 2024-04-22T21:39:12.665384+0000 (oldest deadline 2024-04-22T21:39:37.866187+0000) 2024-04-22T21:39:44.344 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:44 smithi172 ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7[37224]: 2024-04-22T21:39:44.003+0000 7f8ee6162700 -1 osd.7 69 heartbeat_check: no reply from 172.21.15.172:6804 osd.4 since back 2024-04-22T21:39:17.866633+0000 front 2024-04-22T21:39:17.866659+0000 (oldest deadline 2024-04-22T21:39:41.966720+0000) 2024-04-22T21:39:44.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.40 MB/sec execute 78 sec latency 61137.333 ms 2024-04-22T21:39:45.244 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:44 smithi172 podman[40741]: 2024-04-22 21:39:44.956340348 +0000 UTC m=+5.501963573 container died 80378e864f5e0da0f3340194f09d75223b5a1577c9f545bbaecc93176a45af81 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., vcs-type=git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, release=754, GIT_CLEAN=True, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, name=centos-stream, ceph=True) 2024-04-22T21:39:45.593 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:45 smithi172 podman[40741]: 2024-04-22 21:39:45.362745337 +0000 UTC m=+5.908368527 container cleanup 80378e864f5e0da0f3340194f09d75223b5a1577c9f545bbaecc93176a45af81 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vcs-type=git, build-date=2022-05-03T08:36:31.336870, version=8, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, release=754, io.buildah.version=1.19.8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T21:39:45.594 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:45 smithi172 bash[40741]: ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7 2024-04-22T21:39:45.594 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:45 smithi172 bash[40773]: Error: no container with name or ID "ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-22T21:39:45.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.35 MB/sec execute 79 sec latency 62137.369 ms 2024-04-22T21:39:45.854 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:45 smithi172 podman[40760]: 2024-04-22 21:39:45.613754194 +0000 UTC m=+0.650431351 container remove 80378e864f5e0da0f3340194f09d75223b5a1577c9f545bbaecc93176a45af81 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., RELEASE=HEAD, io.openshift.expose-services=, name=centos-stream, vcs-type=git, version=8, GIT_CLEAN=True, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T21:39:45.854 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:45 smithi172 podman[40840]: 2024-04-22 21:39:45.754814724 +0000 UTC m=+0.019048801 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:39:45.854 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:45 smithi172 podman[40840]: 2024-04-22 21:39:45.87292756 +0000 UTC m=+0.137161621 container create 9a4c580c037d3e9ecb604cf471ba6a4c208cd8f71ddf9e07f7a36aecdf8cd871 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7-deactivate, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, release=754, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git) 2024-04-22T21:39:46.147 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:46 smithi172 podman[40840]: 2024-04-22 21:39:46.083648533 +0000 UTC m=+0.347882598 container init 9a4c580c037d3e9ecb604cf471ba6a4c208cd8f71ddf9e07f7a36aecdf8cd871 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, version=8, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_CLEAN=True, ceph=True, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.buildah.version=1.19.8, release=754) 2024-04-22T21:39:46.148 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:46 smithi172 podman[40840]: 2024-04-22 21:39:46.086894494 +0000 UTC m=+0.351128556 container start 9a4c580c037d3e9ecb604cf471ba6a4c208cd8f71ddf9e07f7a36aecdf8cd871 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, release=754, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, vcs-type=git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T21:39:46.148 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:46 smithi172 podman[40840]: 2024-04-22 21:39:46.166780532 +0000 UTC m=+0.431014605 container attach 9a4c580c037d3e9ecb604cf471ba6a4c208cd8f71ddf9e07f7a36aecdf8cd871 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, vendor=Red Hat, Inc., release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, version=8, distribution-scope=public) 2024-04-22T21:39:46.593 INFO:journalctl@ceph.osd.7.smithi172.stdout:Apr 22 21:39:46 smithi172 podman[40840]: 2024-04-22 21:39:46.372758397 +0000 UTC m=+0.636992464 container died 9a4c580c037d3e9ecb604cf471ba6a4c208cd8f71ddf9e07f7a36aecdf8cd871 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_CLEAN=True, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vendor=Red Hat, Inc., vcs-type=git, release=754, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-22T21:39:46.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.31 MB/sec execute 80 sec latency 63137.394 ms 2024-04-22T21:39:46.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3a5a1c14-00ef-11ef-bc93-c7b262605968@osd.7.service' 2024-04-22T21:39:47.003 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:39:47.004 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-22T21:39:47.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T21:39:47.692 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.27 MB/sec execute 81 sec latency 64137.423 ms 2024-04-22T21:39:48.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.23 MB/sec execute 82 sec latency 65137.467 ms 2024-04-22T21:39:49.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.19 MB/sec execute 83 sec latency 66137.517 ms 2024-04-22T21:39:50.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.15 MB/sec execute 84 sec latency 67137.546 ms 2024-04-22T21:39:51.682 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.12 MB/sec execute 85 sec latency 68137.592 ms 2024-04-22T21:39:52.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.08 MB/sec execute 86 sec latency 69137.639 ms 2024-04-22T21:39:53.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.05 MB/sec execute 87 sec latency 70137.687 ms 2024-04-22T21:39:54.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 3.01 MB/sec execute 88 sec latency 71137.739 ms 2024-04-22T21:39:55.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.98 MB/sec execute 89 sec latency 72137.797 ms 2024-04-22T21:39:56.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.94 MB/sec execute 90 sec latency 73137.840 ms 2024-04-22T21:39:57.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.91 MB/sec execute 91 sec latency 74137.879 ms 2024-04-22T21:39:58.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.88 MB/sec execute 92 sec latency 75137.900 ms 2024-04-22T21:39:59.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.85 MB/sec execute 93 sec latency 76137.942 ms 2024-04-22T21:40:00.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.82 MB/sec execute 94 sec latency 77137.986 ms 2024-04-22T21:40:01.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.79 MB/sec execute 95 sec latency 78138.008 ms 2024-04-22T21:40:02.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.76 MB/sec execute 96 sec latency 79138.064 ms 2024-04-22T21:40:03.686 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.73 MB/sec execute 97 sec latency 80138.107 ms 2024-04-22T21:40:04.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.70 MB/sec execute 98 sec latency 81138.145 ms 2024-04-22T21:40:05.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.68 MB/sec execute 99 sec latency 82138.183 ms 2024-04-22T21:40:06.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.65 MB/sec execute 100 sec latency 83138.240 ms 2024-04-22T21:40:07.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.62 MB/sec execute 101 sec latency 84138.267 ms 2024-04-22T21:40:08.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.60 MB/sec execute 102 sec latency 85138.305 ms 2024-04-22T21:40:09.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.57 MB/sec execute 103 sec latency 86138.331 ms 2024-04-22T21:40:10.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.55 MB/sec execute 104 sec latency 87138.345 ms 2024-04-22T21:40:11.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.52 MB/sec execute 105 sec latency 88138.361 ms 2024-04-22T21:40:12.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.50 MB/sec execute 106 sec latency 89138.385 ms 2024-04-22T21:40:13.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T21:40:13.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.48 MB/sec execute 107 sec latency 90138.416 ms 2024-04-22T21:40:14.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.45 MB/sec execute 108 sec latency 91138.453 ms 2024-04-22T21:40:15.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.43 MB/sec execute 109 sec latency 92138.470 ms 2024-04-22T21:40:16.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.41 MB/sec execute 110 sec latency 93138.486 ms 2024-04-22T21:40:17.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.39 MB/sec execute 111 sec latency 94138.535 ms 2024-04-22T21:40:18.683 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.37 MB/sec execute 112 sec latency 95138.623 ms 2024-04-22T21:40:19.684 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.35 MB/sec execute 113 sec latency 96138.744 ms 2024-04-22T21:40:20.684 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.32 MB/sec execute 114 sec latency 97138.845 ms 2024-04-22T21:40:21.684 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.30 MB/sec execute 115 sec latency 98138.947 ms 2024-04-22T21:40:22.684 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.28 MB/sec execute 116 sec latency 99139.068 ms 2024-04-22T21:40:23.684 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.27 MB/sec execute 117 sec latency 100139.145 ms 2024-04-22T21:40:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.25 MB/sec execute 118 sec latency 101139.247 ms 2024-04-22T21:40:25.684 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.23 MB/sec execute 119 sec latency 102139.330 ms 2024-04-22T21:40:26.684 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.21 MB/sec execute 120 sec latency 103139.415 ms 2024-04-22T21:40:27.684 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.19 MB/sec execute 121 sec latency 104139.479 ms 2024-04-22T21:40:28.684 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.17 MB/sec execute 122 sec latency 105139.513 ms 2024-04-22T21:40:29.684 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.15 MB/sec execute 123 sec latency 106139.595 ms 2024-04-22T21:40:30.685 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.14 MB/sec execute 124 sec latency 107139.685 ms 2024-04-22T21:40:31.685 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.12 MB/sec execute 125 sec latency 108139.776 ms 2024-04-22T21:40:32.685 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.10 MB/sec execute 126 sec latency 109139.823 ms 2024-04-22T21:40:33.133 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:40:33.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:40:33.191 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T21:40:33.193 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751/remote/smithi083/crash 2024-04-22T21:40:33.194 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/crash -- . 2024-04-22T21:40:33.246 INFO:teuthology.orchestra.run.smithi083.stderr:tar: /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T21:40:33.247 INFO:teuthology.orchestra.run.smithi083.stderr:tar: Error is not recoverable: exiting now 2024-04-22T21:40:33.248 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751/remote/smithi172/crash 2024-04-22T21:40:33.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/crash -- . 2024-04-22T21:40:33.280 INFO:teuthology.orchestra.run.smithi172.stderr:tar: /var/lib/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T21:40:33.281 INFO:teuthology.orchestra.run.smithi172.stderr:tar: Error is not recoverable: exiting now 2024-04-22T21:40:33.281 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T21:40:33.282 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-22T21:40:33.316 INFO:tasks.cephadm:Compressing logs... 2024-04-22T21:40:33.316 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T21:40:33.359 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T21:40:33.384 INFO:teuthology.orchestra.run.smithi083.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mgr.smithi083.ochghz.log 2024-04-22T21:40:33.384 INFO:teuthology.orchestra.run.smithi083.stderr:: No such file or directory 2024-04-22T21:40:33.384 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.log 2024-04-22T21:40:33.385 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T21:40:33.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mgr.smithi083.ochghz.log: /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T21:40:33.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T21:40:33.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T21:40:33.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.3.log: 86.8% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T21:40:33.385 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T21:40:33.385 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mon.smithi083.log 2024-04-22T21:40:33.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.1.log: /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T21:40:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mon.smithi083.log: gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi083.nyheuj.log 2024-04-22T21:40:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log: 90.8% 83.1% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T21:40:33.390 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T21:40:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T21:40:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi083.nyheuj.log: gzip -5 --verbose 80.8% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi083.nyheuj.log.gz 2024-04-22T21:40:33.390 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/cephadm.log 2024-04-22T21:40:33.394 INFO:teuthology.orchestra.run.smithi172.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mgr.smithi172.edmjti.log 2024-04-22T21:40:33.394 INFO:teuthology.orchestra.run.smithi172.stderr:: No such file or directory 2024-04-22T21:40:33.394 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.log 2024-04-22T21:40:33.394 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T21:40:33.394 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mgr.smithi172.edmjti.log: /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mon.smithi172.log 2024-04-22T21:40:33.394 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-22T21:40:33.395 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T21:40:33.395 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mon.smithi172.log: /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T21:40:33.395 INFO:teuthology.orchestra.run.smithi172.stderr: 86.7% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T21:40:33.395 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi172.cavhbk.log 2024-04-22T21:40:33.395 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-22T21:40:33.395 INFO:teuthology.orchestra.run.smithi172.stderr: 91.0% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T21:40:33.395 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi172.cavhbk.log: gzip -5 60.7% --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T21:40:33.395 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi172.cavhbk.log.gz 2024-04-22T21:40:33.395 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T21:40:33.396 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T21:40:33.396 INFO:teuthology.orchestra.run.smithi172.stderr: 82.1% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T21:40:33.396 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-volume.log: 92.3% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mgr.smithi172.edmjti.log.gz 2024-04-22T21:40:33.396 INFO:teuthology.orchestra.run.smithi172.stderr: 90.7% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T21:40:33.397 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 90.9% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T21:40:33.398 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cephadm.log: 90.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T21:40:33.400 INFO:teuthology.orchestra.run.smithi083.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T21:40:33.482 INFO:teuthology.orchestra.run.smithi172.stderr: 92.6% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mon.smithi172.log.gz 2024-04-22T21:40:33.495 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mgr.smithi083.ochghz.log.gz 2024-04-22T21:40:33.685 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.09 MB/sec execute 127 sec latency 110139.863 ms 2024-04-22T21:40:34.030 INFO:teuthology.orchestra.run.smithi083.stderr: 91.9% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-mon.smithi083.log.gz 2024-04-22T21:40:34.685 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.07 MB/sec execute 128 sec latency 111139.901 ms 2024-04-22T21:40:35.665 INFO:teuthology.orchestra.run.smithi083.stderr: 94.5% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T21:40:35.685 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.05 MB/sec execute 129 sec latency 112139.956 ms 2024-04-22T21:40:36.685 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.04 MB/sec execute 130 sec latency 113140.009 ms 2024-04-22T21:40:37.437 INFO:teuthology.orchestra.run.smithi172.stderr: 94.3% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-22T21:40:37.463 INFO:teuthology.orchestra.run.smithi172.stderr: 94.1% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T21:40:37.685 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.02 MB/sec execute 131 sec latency 114140.079 ms 2024-04-22T21:40:38.195 INFO:teuthology.orchestra.run.smithi083.stderr: 94.3% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T21:40:38.206 INFO:teuthology.orchestra.run.smithi083.stderr: 94.2% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T21:40:38.275 INFO:teuthology.orchestra.run.smithi083.stderr: 94.1% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T21:40:38.277 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-04-22T21:40:38.277 INFO:teuthology.orchestra.run.smithi083.stderr:real 0m4.905s 2024-04-22T21:40:38.277 INFO:teuthology.orchestra.run.smithi083.stderr:user 0m16.920s 2024-04-22T21:40:38.277 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m0.669s 2024-04-22T21:40:38.594 INFO:teuthology.orchestra.run.smithi172.stderr: 94.3% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T21:40:38.685 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 2.01 MB/sec execute 132 sec latency 115140.155 ms 2024-04-22T21:40:39.685 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.99 MB/sec execute 133 sec latency 116140.269 ms 2024-04-22T21:40:40.685 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.98 MB/sec execute 134 sec latency 117140.346 ms 2024-04-22T21:40:40.719 INFO:teuthology.orchestra.run.smithi172.stderr: 94.2% -- replaced with /var/log/ceph/3a5a1c14-00ef-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-22T21:40:40.722 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T21:40:40.722 INFO:teuthology.orchestra.run.smithi172.stderr:real 0m7.349s 2024-04-22T21:40:40.722 INFO:teuthology.orchestra.run.smithi172.stderr:user 0m19.980s 2024-04-22T21:40:40.722 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m0.805s 2024-04-22T21:40:40.722 INFO:tasks.cephadm:Archiving logs... 2024-04-22T21:40:40.723 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751/remote/smithi083/log 2024-04-22T21:40:40.724 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T21:40:41.714 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.96 MB/sec execute 135 sec latency 118140.402 ms 2024-04-22T21:40:42.715 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.95 MB/sec execute 136 sec latency 119140.449 ms 2024-04-22T21:40:43.080 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751/remote/smithi172/log 2024-04-22T21:40:43.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T21:40:43.698 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.93 MB/sec execute 137 sec latency 120140.529 ms 2024-04-22T21:40:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.92 MB/sec execute 138 sec latency 121140.607 ms 2024-04-22T21:40:45.701 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.91 MB/sec execute 139 sec latency 122140.725 ms 2024-04-22T21:40:46.057 INFO:tasks.cephadm:Removing cluster... 2024-04-22T21:40:46.058 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 --force 2024-04-22T21:40:46.361 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 --force 2024-04-22T21:40:46.658 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T21:40:46.658 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T21:40:46.674 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T21:40:46.686 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.89 MB/sec execute 140 sec latency 123140.844 ms 2024-04-22T21:40:46.689 INFO:tasks.cephadm:Teardown complete 2024-04-22T21:40:46.689 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T21:40:46.702 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T21:40:46.703 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T21:40:46.706 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T21:40:46.717 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-04-22T21:40:46.721 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T21:40:46.721 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-04-22T21:40:46.721 INFO:teuthology.orchestra.run.smithi083.stdout:^- hv01.front.sepia.ceph.com 2 6 377 34 +1704us[+1648us] +/- 584ms 2024-04-22T21:40:46.722 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -390us[ -446us] +/- 50ms 2024-04-22T21:40:46.722 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv03.front.sepia.ceph.com 3 6 377 33 +460us[ +404us] +/- 61ms 2024-04-22T21:40:46.722 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:40:46.742 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-04-22T21:40:46.747 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T21:40:46.747 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-04-22T21:40:46.748 INFO:teuthology.orchestra.run.smithi172.stdout:^- hv01.front.sepia.ceph.com 2 6 377 33 +1638us[+1638us] +/- 584ms 2024-04-22T21:40:46.748 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 -478us[ -478us] +/- 50ms 2024-04-22T21:40:46.748 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv03.front.sepia.ceph.com 3 6 377 33 +388us[ +409us] +/- 61ms 2024-04-22T21:40:46.748 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:40:46.748 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T21:40:46.761 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T21:40:46.763 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T21:40:46.777 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T21:40:46.843 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T21:40:46.904 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T21:40:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:40:46.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T21:40:46.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:40:46.958 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T21:40:46.970 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T21:40:46.983 INFO:teuthology.task.internal:Duration was 963.038458 seconds 2024-04-22T21:40:46.983 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T21:40:46.999 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T21:40:47.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T21:40:47.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T21:40:47.036 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T21:40:47.040 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T21:40:47.384 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T21:40:47.385 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-04-22T21:40:47.385 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T21:40:47.405 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T21:40:47.406 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T21:40:47.448 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T21:40:47.448 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T21:40:47.451 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T21:40:47.513 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T21:40:47.513 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T21:40:47.520 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T21:40:47.686 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.88 MB/sec execute 141 sec latency 124140.911 ms 2024-04-22T21:40:48.686 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.87 MB/sec execute 142 sec latency 125140.944 ms 2024-04-22T21:40:49.686 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.85 MB/sec execute 143 sec latency 126140.974 ms 2024-04-22T21:40:50.686 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.84 MB/sec execute 144 sec latency 127141.023 ms 2024-04-22T21:40:51.686 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.83 MB/sec execute 145 sec latency 128141.074 ms 2024-04-22T21:40:52.686 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.82 MB/sec execute 146 sec latency 129141.119 ms 2024-04-22T21:40:52.745 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T21:40:52.755 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T21:40:52.756 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T21:40:52.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T21:40:52.808 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T21:40:52.817 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T21:40:52.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T21:40:52.841 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-04-22T21:40:52.874 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-04-22T21:40:52.910 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T21:40:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:40:52.925 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T21:40:52.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:40:52.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T21:40:52.978 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T21:40:52.980 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751/remote/smithi083 2024-04-22T21:40:52.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T21:40:53.027 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751/remote/smithi172 2024-04-22T21:40:53.027 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T21:40:53.065 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T21:40:53.065 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T21:40:53.067 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T21:40:53.115 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T21:40:53.129 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T21:40:53.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T21:40:53.145 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T21:40:53.145 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T21:40:53.149 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T21:40:53.160 INFO:teuthology.orchestra.run.smithi083.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 21:40 /home/ubuntu/cephtest 2024-04-22T21:40:53.171 INFO:teuthology.orchestra.run.smithi172.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 21:40 /home/ubuntu/cephtest 2024-04-22T21:40:53.172 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T21:40:53.183 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T21:40:53.207 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T21:40:53.237 DEBUG:teuthology.nuke:shortname: smithi083 2024-04-22T21:40:53.237 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T21:40:53.264 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:19:09.111611', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJkiwdunrL3ypsH8NjMiQ8PcUtXSDIJO735kDxInWme+Xmszh8o6zvZNQXDtCDYozAkBA/wKhluGoJkN5ljbcJQ='} 2024-04-22T21:40:53.268 DEBUG:teuthology.nuke:shortname: smithi172 2024-04-22T21:40:53.268 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T21:40:53.293 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668751', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:19:09.113248', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH3I00IjvBBGG1AjUkXfAHZHTfGaFHjiN4OLYSrQkywb/izUmEIwfuYwh5cAo/xJadCM4yDX+ZrkjyP1OFLX4l8='} 2024-04-22T21:40:53.321 INFO:teuthology.orchestra.console.smithi083:Power off 2024-04-22T21:40:53.321 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T21:40:53.346 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2024-04-22T21:40:53.346 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:40:53.365 INFO:teuthology.orchestra.console.smithi172:Power off 2024-04-22T21:40:53.365 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T21:40:53.375 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-04-22T21:40:53.389 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2024-04-22T21:40:53.389 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:40:53.412 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-04-22T21:40:53.686 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.80 MB/sec execute 147 sec latency 130141.193 ms 2024-04-22T21:40:54.686 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.79 MB/sec execute 148 sec latency 131141.271 ms 2024-04-22T21:40:55.686 INFO:teuthology.orchestra.run.smithi083.stdout: 5 85175 1.78 MB/sec execute 149 sec latency 132141.324 ms 2024-04-22T21:40:56.557 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:40:57.377 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:40:57.413 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:40:57.488 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-04-22T21:40:57.523 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-04-22T21:41:01.490 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:41:01.524 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:41:01.601 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2024-04-22T21:41:01.638 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2024-04-22T21:41:01.702 INFO:teuthology.orchestra.console.smithi083:Power off completed 2024-04-22T21:41:01.739 INFO:teuthology.orchestra.console.smithi172:Power off completed 2024-04-22T21:41:01.844 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 963.038458108902 failure_reason: 'Command failed on smithi083 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a5a1c14-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \''ceph versions | jq -e \''"\''"\''.overall | keys\''"\''"\'' | grep $sha1\''''' owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f30207610fa44cdeb43ea797947eb85d status: fail success: false 2024-04-22T21:41:01.845 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T21:41:01.926 INFO:teuthology.run:FAIL