2024-04-22T22:11:34.603 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T22:11:34.603 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --archive /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847 -- /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847/orig.config.yaml 2024-04-22T22:11:34.640 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T22:11:34.724 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7668847' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 207 sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f sleep_before_teardown: 0 subset: 24/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9 targets: smithi047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLACz67ci4U+6RaRmz34a4uSK2gsOM9VQI/yt1dsB7TAg6+kAmT2vpjOzwwksI0Rn331pohypAOrGQD6YegSLDk= smithi113.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCFuda/Xe+k4NML1BaUWlkaIhBmK2uzlS+3ShjPyv66TxnQL0e31CxtKAbjMJXKZ2ROKJX8y6D0TwL6fbYWF/Zc= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-04-22T22:11:34.724 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa; will attempt to use it 2024-04-22T22:11:34.725 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks 2024-04-22T22:11:34.725 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T22:11:34.727 INFO:teuthology.task.internal:Checking packages... 2024-04-22T22:11:34.750 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '430e09df97c8fc7dc2b2ae424f68ed11366c540f' 2024-04-22T22:11:34.751 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T22:11:34.751 INFO:teuthology.packaging:ref: None 2024-04-22T22:11:34.751 INFO:teuthology.packaging:tag: None 2024-04-22T22:11:34.751 INFO:teuthology.packaging:branch: squid 2024-04-22T22:11:34.751 INFO:teuthology.packaging:sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:11:34.751 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-22T22:11:34.991 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2420.g246d7127 2024-04-22T22:11:34.993 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T22:11:35.000 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T22:11:35.000 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T22:11:35.007 INFO:teuthology.task.internal:Saving configuration 2024-04-22T22:11:35.021 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T22:11:35.027 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T22:11:35.052 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 22:05:51.277827', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLACz67ci4U+6RaRmz34a4uSK2gsOM9VQI/yt1dsB7TAg6+kAmT2vpjOzwwksI0Rn331pohypAOrGQD6YegSLDk='} 2024-04-22T22:11:35.078 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 22:05:51.276490', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCFuda/Xe+k4NML1BaUWlkaIhBmK2uzlS+3ShjPyv66TxnQL0e31CxtKAbjMJXKZ2ROKJX8y6D0TwL6fbYWF/Zc='} 2024-04-22T22:11:35.078 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T22:11:35.084 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-22T22:11:35.085 INFO:teuthology.task.internal:roles: ubuntu@smithi113.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-22T22:11:35.085 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T22:11:35.163 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f25c2218f70>, signals=[15]) 2024-04-22T22:11:35.163 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T22:11:35.169 INFO:teuthology.task.internal:Opening connections... 2024-04-22T22:11:35.169 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2024-04-22T22:11:35.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T22:11:35.250 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi113.front.sepia.ceph.com 2024-04-22T22:11:35.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T22:11:35.323 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T22:11:35.330 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2024-04-22T22:11:35.348 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2024-04-22T22:11:35.349 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2024-04-22T22:11:35.407 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="CentOS Stream" 2024-04-22T22:11:35.407 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="9" 2024-04-22T22:11:35.407 INFO:teuthology.orchestra.run.smithi047.stdout:ID="centos" 2024-04-22T22:11:35.408 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE="rhel fedora" 2024-04-22T22:11:35.408 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="9" 2024-04-22T22:11:35.408 INFO:teuthology.orchestra.run.smithi047.stdout:PLATFORM_ID="platform:el9" 2024-04-22T22:11:35.408 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T22:11:35.408 INFO:teuthology.orchestra.run.smithi047.stdout:ANSI_COLOR="0;31" 2024-04-22T22:11:35.408 INFO:teuthology.orchestra.run.smithi047.stdout:LOGO="fedora-logo-icon" 2024-04-22T22:11:35.408 INFO:teuthology.orchestra.run.smithi047.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T22:11:35.408 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://centos.org/" 2024-04-22T22:11:35.408 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T22:11:35.408 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T22:11:35.408 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T22:11:35.409 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2024-04-22T22:11:35.433 DEBUG:teuthology.orchestra.run.smithi113:> uname -m 2024-04-22T22:11:35.450 INFO:teuthology.orchestra.run.smithi113.stdout:x86_64 2024-04-22T22:11:35.451 DEBUG:teuthology.orchestra.run.smithi113:> cat /etc/os-release 2024-04-22T22:11:35.507 INFO:teuthology.orchestra.run.smithi113.stdout:NAME="CentOS Stream" 2024-04-22T22:11:35.507 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION="9" 2024-04-22T22:11:35.507 INFO:teuthology.orchestra.run.smithi113.stdout:ID="centos" 2024-04-22T22:11:35.507 INFO:teuthology.orchestra.run.smithi113.stdout:ID_LIKE="rhel fedora" 2024-04-22T22:11:35.507 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION_ID="9" 2024-04-22T22:11:35.507 INFO:teuthology.orchestra.run.smithi113.stdout:PLATFORM_ID="platform:el9" 2024-04-22T22:11:35.507 INFO:teuthology.orchestra.run.smithi113.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T22:11:35.507 INFO:teuthology.orchestra.run.smithi113.stdout:ANSI_COLOR="0;31" 2024-04-22T22:11:35.507 INFO:teuthology.orchestra.run.smithi113.stdout:LOGO="fedora-logo-icon" 2024-04-22T22:11:35.507 INFO:teuthology.orchestra.run.smithi113.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T22:11:35.508 INFO:teuthology.orchestra.run.smithi113.stdout:HOME_URL="https://centos.org/" 2024-04-22T22:11:35.508 INFO:teuthology.orchestra.run.smithi113.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T22:11:35.508 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T22:11:35.508 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T22:11:35.508 INFO:teuthology.lock.ops:Updating smithi113.front.sepia.ceph.com on lock server 2024-04-22T22:11:35.532 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T22:11:35.541 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T22:11:35.545 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T22:11:35.546 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2024-04-22T22:11:35.548 DEBUG:teuthology.orchestra.run.smithi113:> test '!' -e /home/ubuntu/cephtest 2024-04-22T22:11:35.564 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T22:11:35.570 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T22:11:35.570 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2024-04-22T22:11:35.606 DEBUG:teuthology.orchestra.run.smithi113:> test -z $(ls -A /var/lib/ceph) 2024-04-22T22:11:35.638 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T22:11:35.694 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T22:11:35.724 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T22:11:35.724 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T22:11:35.724 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T22:11:35.725 DEBUG:teuthology.orchestra.run.smithi113:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T22:11:35.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:11:35.741 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2024-04-22T22:11:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:11:35.743 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-04-22T22:11:35.797 INFO:teuthology.orchestra.run.smithi113.stdout:5.14.0-437.el9.x86_64 2024-04-22T22:11:35.797 INFO:teuthology.task.kernel:Running kernel on smithi113: 5.14.0-437.el9.x86_64 2024-04-22T22:11:35.798 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum install -y kernel 2024-04-22T22:11:35.800 INFO:teuthology.orchestra.run.smithi047.stdout:5.14.0-437.el9.x86_64 2024-04-22T22:11:35.800 INFO:teuthology.task.kernel:Running kernel on smithi047: 5.14.0-437.el9.x86_64 2024-04-22T22:11:35.801 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum install -y kernel 2024-04-22T22:11:38.980 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.0 MB 00:01 2024-04-22T22:11:40.499 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 8.0 MB 00:03 2024-04-22T22:11:40.882 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-22T22:11:43.358 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-22T22:11:45.351 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.2 MB 00:00 2024-04-22T22:11:47.700 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-04-22T22:11:48.945 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-22T22:11:49.825 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.2 MB 00:02 2024-04-22T22:11:51.284 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 16 kB 00:00 2024-04-22T22:11:52.446 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-22T22:11:54.242 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-04-22T22:11:56.232 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T22:11:56.232 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T22:11:56.338 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-04-22T22:11:56.339 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-04-22T22:11:56.339 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-04-22T22:11:56.462 DEBUG:teuthology.orchestra.run.smithi047:> echo no | sudo yum reinstall kernel || true 2024-04-22T22:11:57.024 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 10:11:54 PM UTC. 2024-04-22T22:11:57.166 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-04-22T22:11:57.172 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-04-22T22:11:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-04-22T22:11:57.172 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-04-22T22:11:57.173 INFO:teuthology.orchestra.run.smithi047.stdout:Reinstalling: 2024-04-22T22:11:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T22:11:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:11:57.173 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-04-22T22:11:57.173 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-04-22T22:11:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:11:57.173 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 255 k 2024-04-22T22:11:57.174 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 0 2024-04-22T22:11:57.174 INFO:teuthology.orchestra.run.smithi047.stderr:Operation aborted. 2024-04-22T22:11:57.237 INFO:teuthology.orchestra.run.smithi047.stdout:Is this ok [y/N]: 2024-04-22T22:11:57.237 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum reinstall -y kernel || true 2024-04-22T22:11:57.459 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-04-22T22:11:57.757 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 10:11:54 PM UTC. 2024-04-22T22:11:57.877 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-04-22T22:11:57.878 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-04-22T22:11:57.878 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-04-22T22:11:57.878 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-04-22T22:11:57.878 INFO:teuthology.orchestra.run.smithi047.stdout:Reinstalling: 2024-04-22T22:11:57.878 INFO:teuthology.orchestra.run.smithi047.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T22:11:57.878 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:11:57.879 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-04-22T22:11:57.879 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-04-22T22:11:57.879 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:11:57.879 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 255 k 2024-04-22T22:11:57.879 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 0 2024-04-22T22:11:57.879 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-04-22T22:11:58.469 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-5.14.0-437.el9.x86_64.rpm 484 kB/s | 255 kB 00:00 2024-04-22T22:11:58.470 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-04-22T22:11:58.470 INFO:teuthology.orchestra.run.smithi047.stdout:Total 431 kB/s | 255 kB 00:00 2024-04-22T22:11:58.491 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-04-22T22:11:58.676 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-04-22T22:11:58.676 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-04-22T22:11:58.681 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-04-22T22:11:58.681 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-04-22T22:11:58.828 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-04-22T22:11:58.920 INFO:teuthology.orchestra.run.smithi047.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T22:11:58.921 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T22:11:59.290 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T22:11:59.290 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T22:11:59.384 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-22T22:11:59.385 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-04-22T22:11:59.385 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-22T22:11:59.498 DEBUG:teuthology.orchestra.run.smithi113:> echo no | sudo yum reinstall kernel || true 2024-04-22T22:11:59.589 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T22:11:59.589 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T22:12:00.041 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 10:11:57 PM UTC. 2024-04-22T22:12:00.203 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-22T22:12:00.207 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-22T22:12:00.208 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-22T22:12:00.208 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-22T22:12:00.208 INFO:teuthology.orchestra.run.smithi113.stdout:Reinstalling: 2024-04-22T22:12:00.208 INFO:teuthology.orchestra.run.smithi113.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T22:12:00.208 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:12:00.208 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-22T22:12:00.208 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-22T22:12:00.208 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:12:00.208 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 255 k 2024-04-22T22:12:00.208 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 0 2024-04-22T22:12:00.208 INFO:teuthology.orchestra.run.smithi113.stderr:Operation aborted. 2024-04-22T22:12:00.274 INFO:teuthology.orchestra.run.smithi113.stdout:Is this ok [y/N]: 2024-04-22T22:12:00.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum reinstall -y kernel || true 2024-04-22T22:12:00.317 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T22:12:00.317 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:12:00.317 INFO:teuthology.orchestra.run.smithi047.stdout:Reinstalled: 2024-04-22T22:12:00.317 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T22:12:00.317 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:12:00.317 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-04-22T22:12:00.509 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T22:12:00.558 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T22:12:00.558 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T22:12:00.558 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T22:12:00.559 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-22T22:12:00.559 DEBUG:teuthology.orchestra.run.smithi047:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T22:12:00.593 INFO:teuthology.orchestra.run.smithi047.stdout:ttyS1 2024-04-22T22:12:00.611 DEBUG:teuthology.parallel:result is None 2024-04-22T22:12:00.764 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 10:11:57 PM UTC. 2024-04-22T22:12:00.880 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout:Reinstalling: 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 255 k 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 0 2024-04-22T22:12:00.881 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-22T22:12:01.395 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-22T22:12:01.395 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-22T22:12:01.395 INFO:teuthology.orchestra.run.smithi113.stdout:Total 496 kB/s | 255 kB 00:00 2024-04-22T22:12:01.410 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-22T22:12:01.590 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-22T22:12:01.590 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-22T22:12:01.595 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-22T22:12:01.595 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-22T22:12:01.742 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-22T22:12:01.829 INFO:teuthology.orchestra.run.smithi113.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T22:12:01.829 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T22:12:02.536 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T22:12:02.536 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T22:12:03.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T22:12:03.231 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:12:03.231 INFO:teuthology.orchestra.run.smithi113.stdout:Reinstalled: 2024-04-22T22:12:03.231 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T22:12:03.231 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:12:03.231 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-22T22:12:03.398 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T22:12:03.445 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T22:12:03.445 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T22:12:03.445 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T22:12:03.445 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-22T22:12:03.446 DEBUG:teuthology.orchestra.run.smithi113:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T22:12:03.485 INFO:teuthology.orchestra.run.smithi113.stdout:ttyS1 2024-04-22T22:12:03.515 DEBUG:teuthology.parallel:result is None 2024-04-22T22:12:03.515 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T22:12:03.525 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T22:12:03.525 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T22:12:03.528 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T22:12:03.570 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T22:12:03.577 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T22:12:03.585 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T22:12:03.585 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T22:12:03.588 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T22:12:03.644 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T22:12:03.651 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T22:12:03.651 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T22:12:03.655 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T22:12:03.683 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T22:12:03.694 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T22:12:03.712 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T22:12:03.721 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T22:12:03.723 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T22:12:03.730 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T22:12:03.730 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T22:12:03.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T22:12:03.786 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T22:12:03.813 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T22:12:03.813 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T22:12:03.815 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T22:12:03.841 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T22:12:03.912 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T22:12:03.961 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T22:12:04.044 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T22:12:04.086 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-22T22:12:04.086 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T22:12:04.151 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T22:12:04.191 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T22:12:04.243 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T22:12:04.326 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T22:12:04.364 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-22T22:12:04.364 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T22:12:04.428 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2024-04-22T22:12:04.431 DEBUG:teuthology.orchestra.run.smithi113:> sudo service rsyslog restart 2024-04-22T22:12:04.467 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T22:12:04.508 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T22:12:05.018 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T22:12:05.025 INFO:teuthology.task.internal:Starting timer... 2024-04-22T22:12:05.026 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T22:12:05.035 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T22:12:05.043 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-22T22:12:05.044 DEBUG:teuthology.orchestra.run.smithi047:> sudo service auditd rotate 2024-04-22T22:12:05.085 INFO:teuthology.orchestra.run.smithi047.stdout:Rotating logs: 2024-04-22T22:12:05.087 DEBUG:teuthology.orchestra.run.smithi113:> sudo service auditd rotate 2024-04-22T22:12:05.154 INFO:teuthology.orchestra.run.smithi113.stdout:Rotating logs: 2024-04-22T22:12:05.156 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T22:12:05.156 DEBUG:teuthology.orchestra.run.smithi047:> /usr/sbin/getenforce 2024-04-22T22:12:05.187 INFO:teuthology.orchestra.run.smithi047.stdout:Permissive 2024-04-22T22:12:05.188 DEBUG:teuthology.orchestra.run.smithi113:> /usr/sbin/getenforce 2024-04-22T22:12:05.211 INFO:teuthology.orchestra.run.smithi113.stdout:Permissive 2024-04-22T22:12:05.212 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi047.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi113.front.sepia.ceph.com': 'permissive'} 2024-04-22T22:12:05.212 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T22:12:05.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:12:05.254 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T22:12:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:12:05.279 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T22:12:05.280 DEBUG:teuthology.orchestra.run.smithi047:> sudo /usr/sbin/setenforce permissive 2024-04-22T22:12:05.323 DEBUG:teuthology.orchestra.run.smithi113:> sudo /usr/sbin/setenforce permissive 2024-04-22T22:12:05.348 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T22:12:05.357 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T22:12:05.374 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T22:12:05.375 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi047.front.sepia.ceph.com,smithi113.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T22:15:41.997 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi113.front.sepia.ceph.com')] 2024-04-22T22:15:41.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-04-22T22:15:41.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T22:15:42.095 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-04-22T22:15:42.184 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-04-22T22:15:42.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-22T22:15:42.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T22:15:42.259 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-04-22T22:15:42.341 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-22T22:15:42.342 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T22:15:42.353 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T22:15:42.354 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T22:15:42.354 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T22:15:42.357 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T22:15:42.357 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T22:15:42.389 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T22:15:42.405 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T22:15:42.409 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T22:15:42.422 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T22:15:42.430 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: ntpd: command not found 2024-04-22T22:15:42.443 INFO:teuthology.orchestra.run.smithi047.stdout:506 Cannot talk to daemon 2024-04-22T22:15:42.445 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: ntpd: command not found 2024-04-22T22:15:42.455 INFO:teuthology.orchestra.run.smithi113.stdout:506 Cannot talk to daemon 2024-04-22T22:15:42.460 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T22:15:42.468 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T22:15:42.474 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T22:15:42.481 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T22:15:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: ntpq: command not found 2024-04-22T22:15:42.536 INFO:teuthology.orchestra.run.smithi113.stderr:bash: line 1: ntpq: command not found 2024-04-22T22:15:42.538 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T22:15:42.538 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-04-22T22:15:42.538 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:15:42.538 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:15:42.538 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:15:42.538 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:15:42.540 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T22:15:42.540 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-04-22T22:15:42.540 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:15:42.540 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:15:42.540 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:15:42.540 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:15:42.541 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T22:15:42.656 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '430e09df97c8fc7dc2b2ae424f68ed11366c540f'} 2024-04-22T22:15:42.656 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-22T22:15:42.656 INFO:tasks.cephadm:Cluster fsid is dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:15:42.657 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T22:15:42.657 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-22T22:15:42.657 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi047': '172.21.15.47', 'mon.smithi113': '172.21.15.113'} 2024-04-22T22:15:42.657 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T22:15:42.657 DEBUG:teuthology.orchestra.run.smithi047:> sudo hostname $(hostname -s) 2024-04-22T22:15:42.684 DEBUG:teuthology.orchestra.run.smithi113:> sudo hostname $(hostname -s) 2024-04-22T22:15:42.709 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-22T22:15:42.709 DEBUG:teuthology.orchestra.run.smithi047:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T22:15:42.987 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 22 22:15 /home/ubuntu/cephtest/cephadm 2024-04-22T22:15:42.987 DEBUG:teuthology.orchestra.run.smithi113:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T22:15:43.163 INFO:teuthology.orchestra.run.smithi113.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 22 22:15 /home/ubuntu/cephtest/cephadm 2024-04-22T22:15:43.163 DEBUG:teuthology.orchestra.run.smithi047:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T22:15:43.187 DEBUG:teuthology.orchestra.run.smithi113:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T22:15:43.325 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-22T22:15:43.325 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-22T22:15:43.328 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-22T22:15:44.865 INFO:teuthology.orchestra.run.smithi113.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T22:15:44.881 INFO:teuthology.orchestra.run.smithi047.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T22:16:02.561 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-04-22T22:16:02.561 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-22T22:16:02.561 INFO:teuthology.orchestra.run.smithi047.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-22T22:16:02.561 INFO:teuthology.orchestra.run.smithi047.stdout: "repo_digests": [ 2024-04-22T22:16:02.561 INFO:teuthology.orchestra.run.smithi047.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-22T22:16:02.561 INFO:teuthology.orchestra.run.smithi047.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-22T22:16:02.561 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-04-22T22:16:02.562 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-04-22T22:16:03.442 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-22T22:16:03.442 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-22T22:16:03.442 INFO:teuthology.orchestra.run.smithi113.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-22T22:16:03.442 INFO:teuthology.orchestra.run.smithi113.stdout: "repo_digests": [ 2024-04-22T22:16:03.442 INFO:teuthology.orchestra.run.smithi113.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-22T22:16:03.442 INFO:teuthology.orchestra.run.smithi113.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-22T22:16:03.442 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-04-22T22:16:03.442 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-22T22:16:03.460 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph 2024-04-22T22:16:03.494 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /etc/ceph 2024-04-22T22:16:03.521 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /etc/ceph 2024-04-22T22:16:03.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 777 /etc/ceph 2024-04-22T22:16:03.589 INFO:tasks.cephadm:Writing seed config... 2024-04-22T22:16:03.590 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T22:16:03.590 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T22:16:03.590 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T22:16:03.590 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T22:16:03.590 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T22:16:03.590 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T22:16:03.590 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T22:16:03.590 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-22T22:16:03.591 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-22T22:16:03.591 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T22:16:03.621 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = dbcf7660-00f5-11ef-bc93-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-22T22:16:03.622 DEBUG:teuthology.orchestra.run.smithi047:mon.smithi047> sudo journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mon.smithi047.service 2024-04-22T22:16:03.664 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T22:16:03.664 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.47 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:16:03.868 INFO:teuthology.orchestra.run.smithi047.stderr:-------------------------------------------------------------------------------- 2024-04-22T22:16:03.869 INFO:teuthology.orchestra.run.smithi047.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'dbcf7660-00f5-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.47', '--skip-admin-label'] 2024-04-22T22:16:03.893 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: 4.9.4-dev 2024-04-22T22:16:03.895 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying podman|docker is present... 2024-04-22T22:16:03.924 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: 4.9.4-dev 2024-04-22T22:16:03.927 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying lvm2 is present... 2024-04-22T22:16:03.927 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying time synchronization is in place... 2024-04-22T22:16:03.934 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T22:16:03.940 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-04-22T22:16:03.947 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: enabled 2024-04-22T22:16:03.953 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: active 2024-04-22T22:16:03.954 INFO:teuthology.orchestra.run.smithi047.stderr:Unit chronyd.service is enabled and running 2024-04-22T22:16:03.954 INFO:teuthology.orchestra.run.smithi047.stderr:Repeating the final host check... 2024-04-22T22:16:03.976 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: 4.9.4-dev 2024-04-22T22:16:03.978 INFO:teuthology.orchestra.run.smithi047.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-22T22:16:03.978 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl is present 2024-04-22T22:16:03.978 INFO:teuthology.orchestra.run.smithi047.stderr:lvcreate is present 2024-04-22T22:16:03.984 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T22:16:03.990 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-04-22T22:16:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: enabled 2024-04-22T22:16:04.003 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: active 2024-04-22T22:16:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:Unit chronyd.service is enabled and running 2024-04-22T22:16:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:Host looks OK 2024-04-22T22:16:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:Cluster fsid: dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:16:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:Acquiring lock 140433820848864 on /run/cephadm/dbcf7660-00f5-11ef-bc93-c7b262605968.lock 2024-04-22T22:16:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:Lock 140433820848864 acquired on /run/cephadm/dbcf7660-00f5-11ef-bc93-c7b262605968.lock 2024-04-22T22:16:04.005 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying IP 172.21.15.47 port 3300 ... 2024-04-22T22:16:04.005 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying IP 172.21.15.47 port 6789 ... 2024-04-22T22:16:04.005 INFO:teuthology.orchestra.run.smithi047.stderr:Base mon IP is 172.21.15.47, final addrv is [v2:172.21.15.47:3300,v1:172.21.15.47:6789] 2024-04-22T22:16:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.47 metric 100 2024-04-22T22:16:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.47 metric 100 2024-04-22T22:16:04.010 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T22:16:04.010 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-22T22:16:04.010 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-22T22:16:04.013 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T22:16:04.013 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-22T22:16:04.013 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-22T22:16:04.013 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-22T22:16:04.013 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:148e/64 scope link noprefixroute 2024-04-22T22:16:04.013 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-22T22:16:04.014 INFO:teuthology.orchestra.run.smithi047.stderr:Mon IP `172.21.15.47` is in CIDR network `172.21.0.0/20` 2024-04-22T22:16:04.014 INFO:teuthology.orchestra.run.smithi047.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T22:16:04.014 INFO:teuthology.orchestra.run.smithi047.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T22:16:04.119 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-22T22:16:04.505 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Getting image source signatures 2024-04-22T22:16:04.505 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-22T22:16:04.505 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-22T22:16:04.505 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-22T22:16:04.505 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-22T22:16:04.505 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-22T22:16:04.505 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-22T22:16:04.671 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Writing manifest to image destination 2024-04-22T22:16:04.825 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-22T22:16:05.199 INFO:teuthology.orchestra.run.smithi047.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-22T22:16:05.723 INFO:teuthology.orchestra.run.smithi047.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-22T22:16:05.723 INFO:teuthology.orchestra.run.smithi047.stderr:Extracting ceph user uid/gid from container image... 2024-04-22T22:16:06.432 INFO:teuthology.orchestra.run.smithi047.stderr:stat: 167 167 2024-04-22T22:16:12.113 INFO:teuthology.orchestra.run.smithi047.stderr:Creating initial keys... 2024-04-22T22:16:15.666 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-authtool: AQCu4SZm5XamNBAAMM95T5LVNL/aoV3q5s8xHg== 2024-04-22T22:16:18.165 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-authtool: AQCy4SZmGkqaBRAA4rR+8ePf42KfhYxPhQtjLw== 2024-04-22T22:16:19.253 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-authtool: AQCz4SZmN4DaCxAAfEDMRy6vdBajgiOSgpgPDg== 2024-04-22T22:16:19.719 INFO:teuthology.orchestra.run.smithi047.stderr:Creating initial monmap... 2024-04-22T22:16:20.226 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T22:16:20.226 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-22T22:16:20.226 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:16:20.226 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T22:16:20.691 INFO:teuthology.orchestra.run.smithi047.stderr:monmaptool for smithi047 [v2:172.21.15.47:3300,v1:172.21.15.47:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T22:16:20.691 INFO:teuthology.orchestra.run.smithi047.stderr:setting min_mon_release = octopus 2024-04-22T22:16:20.691 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: set fsid to dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:16:20.691 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T22:16:20.691 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-04-22T22:16:20.691 INFO:teuthology.orchestra.run.smithi047.stderr:Creating mon... 2024-04-22T22:16:21.619 INFO:teuthology.orchestra.run.smithi047.stderr:create mon.smithi047 on 2024-04-22T22:16:21.896 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T22:16:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-dbcf7660-00f5-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-dbcf7660-00f5-11ef-bc93-c7b262605968.target. 2024-04-22T22:16:22.046 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-dbcf7660-00f5-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-dbcf7660-00f5-11ef-bc93-c7b262605968.target. 2024-04-22T22:16:22.346 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to reset failed state of unit ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mon.smithi047.service: Unit ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mon.smithi047.service not loaded. 2024-04-22T22:16:22.354 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dbcf7660-00f5-11ef-bc93-c7b262605968.target.wants/ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mon.smithi047.service → /etc/systemd/system/ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@.service. 2024-04-22T22:16:23.127 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:22 smithi047 podman[21485]: 2024-04-22 22:16:22.794246058 +0000 UTC m=+0.018731990 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-22T22:16:23.418 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:23 smithi047 podman[21485]: 2024-04-22 22:16:23.176219841 +0000 UTC m=+0.400705760 container create ff2f201f4310becc2663141158ea87331a9205534ebfd2ab622823544fc0d8e1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, RELEASE=HEAD, architecture=x86_64, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , vcs-type=git) 2024-04-22T22:16:23.418 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:23 smithi047 podman[21485]: 2024-04-22 22:16:23.370133175 +0000 UTC m=+0.594619093 container init ff2f201f4310becc2663141158ea87331a9205534ebfd2ab622823544fc0d8e1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_CLEAN=True, version=8, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.tags=base centos centos-stream) 2024-04-22T22:16:23.418 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:23 smithi047 podman[21485]: 2024-04-22 22:16:23.373599766 +0000 UTC m=+0.598085693 container start ff2f201f4310becc2663141158ea87331a9205534ebfd2ab622823544fc0d8e1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047, name=centos-stream, release=754, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, version=8, vendor=Red Hat, Inc.) 2024-04-22T22:16:23.418 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:23 smithi047 ceph-mon[21504]: mkfs dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:16:23.418 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:23 smithi047 ceph-mon[21504]: mon.smithi047 is new leader, mons smithi047 in quorum (ranks 0) 2024-04-22T22:16:23.418 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:23 smithi047 bash[21485]: ff2f201f4310becc2663141158ea87331a9205534ebfd2ab622823544fc0d8e1 2024-04-22T22:16:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2024-04-22T22:16:23.439 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-04-22T22:16:23.440 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2024-04-22T22:16:23.440 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to enable service . firewalld.service is not available 2024-04-22T22:16:23.440 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mon to start... 2024-04-22T22:16:23.440 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mon... 2024-04-22T22:16:23.673 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:23 smithi047 systemd[1]: Started Ceph mon.smithi047 for dbcf7660-00f5-11ef-bc93-c7b262605968. 2024-04-22T22:16:23.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: cluster: 2024-04-22T22:16:23.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: id: dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:16:23.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-22T22:16:23.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:23.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: services: 2024-04-22T22:16:23.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi047 (age 0.581044s) 2024-04-22T22:16:23.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-22T22:16:23.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-22T22:16:23.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:23.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: data: 2024-04-22T22:16:23.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-22T22:16:23.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-22T22:16:23.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-22T22:16:23.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: pgs: 2024-04-22T22:16:23.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:24.485 INFO:teuthology.orchestra.run.smithi047.stderr:mon is available 2024-04-22T22:16:24.485 INFO:teuthology.orchestra.run.smithi047.stderr:Assimilating anything we can from ceph.conf... 2024-04-22T22:16:24.700 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:24 smithi047 ceph-mon[21504]: mon.smithi047 is new leader, mons smithi047 in quorum (ranks 0) 2024-04-22T22:16:24.700 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:24 smithi047 ceph-mon[21504]: monmap e1: 1 mons at {smithi047=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0]} 2024-04-22T22:16:24.700 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:24 smithi047 ceph-mon[21504]: fsmap 2024-04-22T22:16:24.700 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:24 smithi047 ceph-mon[21504]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T22:16:24.700 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:24 smithi047 ceph-mon[21504]: mgrmap e1: no daemons active 2024-04-22T22:16:24.700 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:24 smithi047 ceph-mon[21504]: from='client.? 172.21.15.47:0/4012477617' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T22:16:24.994 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:24.994 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [global] 2024-04-22T22:16:24.994 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: fsid = dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:16:24.994 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.47:3300,v1:172.21.15.47:6789] 2024-04-22T22:16:24.994 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-22T22:16:24.994 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-22T22:16:24.994 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-22T22:16:24.995 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-22T22:16:24.995 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:24.995 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [mgr] 2024-04-22T22:16:24.995 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-22T22:16:24.995 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:24.995 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [osd] 2024-04-22T22:16:24.995 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-22T22:16:24.995 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-22T22:16:25.582 INFO:teuthology.orchestra.run.smithi047.stderr:Generating new minimal ceph.conf... 2024-04-22T22:16:25.707 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:25 smithi047 ceph-mon[21504]: from='client.? 172.21.15.47:0/3439115774' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T22:16:25.708 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:25 smithi047 ceph-mon[21504]: from='client.? 172.21.15.47:0/3439115774' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T22:16:26.639 INFO:teuthology.orchestra.run.smithi047.stderr:Restarting the monitor... 2024-04-22T22:16:26.663 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:26 smithi047 ceph-mon[21504]: from='client.? 172.21.15.47:0/1459145867' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:16:27.127 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:26 smithi047 systemd[1]: Stopping Ceph mon.smithi047 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:16:27.127 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:26 smithi047 bash[21724]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon.smithi047" found: no such container 2024-04-22T22:16:27.127 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:26 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047[21500]: 2024-04-22T22:16:26.841+0000 7f9be26cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi047 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:16:27.127 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:26 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047[21500]: 2024-04-22T22:16:26.841+0000 7f9be26cb700 -1 mon.smithi047@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T22:16:27.127 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:26 smithi047 podman[21737]: 2024-04-22 22:16:26.933710929 +0000 UTC m=+0.206324704 container died ff2f201f4310becc2663141158ea87331a9205534ebfd2ab622823544fc0d8e1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047, architecture=x86_64, GIT_CLEAN=True, distribution-scope=public, name=centos-stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , release=754) 2024-04-22T22:16:27.531 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:27 smithi047 podman[21737]: 2024-04-22 22:16:27.282228571 +0000 UTC m=+0.554842345 container cleanup ff2f201f4310becc2663141158ea87331a9205534ebfd2ab622823544fc0d8e1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, distribution-scope=public, ceph=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, GIT_CLEAN=True, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream) 2024-04-22T22:16:27.531 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:27 smithi047 bash[21737]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047 2024-04-22T22:16:27.531 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:27 smithi047 bash[21764]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon.smithi047" found: no such container 2024-04-22T22:16:27.782 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:27 smithi047 podman[21750]: 2024-04-22 22:16:27.558070678 +0000 UTC m=+0.622667742 container remove ff2f201f4310becc2663141158ea87331a9205534ebfd2ab622823544fc0d8e1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047, vcs-type=git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, version=8, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, name=centos-stream, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:16:27.782 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:27 smithi047 systemd[1]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mon.smithi047.service: Deactivated successfully. 2024-04-22T22:16:27.782 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:27 smithi047 systemd[1]: Stopped Ceph mon.smithi047 for dbcf7660-00f5-11ef-bc93-c7b262605968. 2024-04-22T22:16:27.782 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:27 smithi047 systemd[1]: Starting Ceph mon.smithi047 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:16:28.050 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:27 smithi047 podman[21827]: 2024-04-22 22:16:27.808753769 +0000 UTC m=+0.112997565 container create 96cda5acb6353787a18bdf57e2fadf57f04f565897236605ed28e7993f67458e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.openshift.expose-services=, release=754, architecture=x86_64, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public) 2024-04-22T22:16:28.050 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:27 smithi047 podman[21827]: 2024-04-22 22:16:27.715007069 +0000 UTC m=+0.019250892 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-22T22:16:28.050 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 podman[21827]: 2024-04-22 22:16:28.002605033 +0000 UTC m=+0.306848835 container init 96cda5acb6353787a18bdf57e2fadf57f04f565897236605ed28e7993f67458e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, name=centos-stream, vcs-type=git, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., architecture=x86_64) 2024-04-22T22:16:28.051 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 podman[21827]: 2024-04-22 22:16:28.005707928 +0000 UTC m=+0.309951726 container start 96cda5acb6353787a18bdf57e2fadf57f04f565897236605ed28e7993f67458e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, distribution-scope=public, vendor=Red Hat, Inc., release=754, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_BRANCH=HEAD, vcs-type=git) 2024-04-22T22:16:28.051 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T22:16:28.051 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-22T22:16:28.051 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: pidfile_write: ignore empty --pid-file 2024-04-22T22:16:28.051 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: load: jerasure load: lrc 2024-04-22T22:16:28.051 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: RocksDB version: 6.15.5 2024-04-22T22:16:28.051 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-22T22:16:28.051 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Compile date Apr 18 2022 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: DB SUMMARY 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: DB Session ID: R1MCZQ2BJJAPS8211BD1 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: CURRENT file: CURRENT 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi047/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi047/store.db: 000010.log size: 82713 ; 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.error_if_exists: 0 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.create_if_missing: 0 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.paranoid_checks: 1 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.env: 0x55a9410c6860 2024-04-22T22:16:28.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.fs: Posix File System 2024-04-22T22:16:28.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.info_log: 0x55a9426f1fa0 2024-04-22T22:16:28.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T22:16:28.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.statistics: (nil) 2024-04-22T22:16:28.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.use_fsync: 0 2024-04-22T22:16:28.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_log_file_size: 0 2024-04-22T22:16:28.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T22:16:28.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T22:16:28.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T22:16:28.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T22:16:28.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.allow_fallocate: 1 2024-04-22T22:16:28.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T22:16:28.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T22:16:28.054 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.use_direct_reads: 0 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.db_log_dir: 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi047/store.db 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T22:16:28.055 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.write_buffer_manager: 0x55a9427e2330 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.unordered_write: 0 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T22:16:28.056 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.row_cache: None 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.wal_filter: None 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.preserve_deletes: 0 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.two_write_queues: 0 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.atomic_flush: 0 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T22:16:28.057 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.log_readahead_size: 0 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_background_jobs: 2 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_background_compactions: -1 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_subcompactions: 1 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T22:16:28.058 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_open_files: -1 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_background_flushes: -1 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Compression algorithms supported: 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T22:16:28.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: kZSTD supported: 0 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: kXpressCompression supported: 0 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: kLZ4Compression supported: 1 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: kBZip2Compression supported: 0 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: kZlibCompression supported: 1 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: kSnappyCompression supported: 1 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi047/store.db/MANIFEST-000009 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.merge_operator: 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_filter: None 2024-04-22T22:16:28.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a9426bdd00) 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: cache_index_and_filter_blocks: 1 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: pin_top_level_index_and_filter: 1 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: index_type: 0 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: data_block_index_type: 0 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: index_shortening: 1 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T22:16:28.061 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: hash_index_allow_collision: 1 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: checksum: 1 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: no_block_cache: 0 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: block_cache: 0x55a942728170 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: block_cache_name: BinnedLRUCache 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: block_cache_options: 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: capacity : 536870912 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: num_shard_bits : 4 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: strict_capacity_limit : 0 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: high_pri_pool_ratio: 0.000 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: block_cache_compressed: (nil) 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: persistent_cache: (nil) 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: block_size: 4096 2024-04-22T22:16:28.062 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: block_size_deviation: 10 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: block_restart_interval: 16 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: index_block_restart_interval: 1 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: metadata_block_size: 4096 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: partition_filters: 0 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: use_delta_encoding: 1 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: whole_key_filtering: 1 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: verify_compression: 0 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: read_amp_bytes_per_bit: 0 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: format_version: 4 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: enable_index_compression: 1 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: block_align: 0 2024-04-22T22:16:28.063 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compression: NoCompression 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.num_levels: 7 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T22:16:28.064 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T22:16:28.065 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T22:16:28.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T22:16:28.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T22:16:28.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T22:16:28.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T22:16:28.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.arena_block_size: 4194304 2024-04-22T22:16:28.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T22:16:28.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T22:16:28.067 INFO:teuthology.orchestra.run.smithi047.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-22T22:16:28.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-22T22:16:28.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T22:16:28.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T22:16:28.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.table_properties_collectors: 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.inplace_update_support: 0 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T22:16:28.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.bloom_locality: 0 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.max_successive_merges: 0 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.ttl: 2592000 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.enable_blob_files: false 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.min_blob_size: 0 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T22:16:28.070 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T22:16:28.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T22:16:28.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T22:16:28.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi047/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T22:16:28.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-22T22:16:28.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-22T22:16:28.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713824188032171, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-22T22:16:28.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-22T22:16:28.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-22T22:16:28.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713824188033037, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79395, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77659, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71968, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713824188, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "35c842c4-b5e7-4a37-9c43-38f5c8cff851", "db_session_id": "R1MCZQ2BJJAPS8211BD1"}} 2024-04-22T22:16:28.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-22T22:16:28.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713824188033377, "job": 1, "event": "recovery_finished"} 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi047/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55a94270ea80 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: DB pointer 0x55a94271e000 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: ** DB Stats ** 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-22T22:16:28.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: ** Compaction Stats [default] ** 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: L0 2/0 79.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.3 0.00 0.00 1 0.000 0 0 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Sum 2/0 79.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.3 0.00 0.00 1 0.000 0 0 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.3 0.00 0.00 1 0.000 0 0 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: ** Compaction Stats [default] ** 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:16:28.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.3 0.00 0.00 1 0.000 0 0 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Cumulative compaction: 0.00 GB write, 24.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Interval compaction: 0.00 GB write, 24.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: 2024-04-22T22:16:28.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: ** Compaction Stats [default] ** 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: L0 2/0 79.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.3 0.00 0.00 1 0.000 0 0 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Sum 2/0 79.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.3 0.00 0.00 1 0.000 0 0 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: ** Compaction Stats [default] ** 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.3 0.00 0.00 1 0.000 0 0 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T22:16:28.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Cumulative compaction: 0.00 GB write, 24.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: starting mon.smithi047 rank 0 at public addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] at bind addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi047 fsid dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: mon.smithi047@-1(???) e1 preinit fsid dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: mon.smithi047@-1(???).mds e1 new map 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: mon.smithi047@-1(???).mds e1 print_map 2024-04-22T22:16:28.076 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: e1 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: legacy client fscid: -1 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout: No filesystems configured 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: mon.smithi047@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: mon.smithi047@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: mon.smithi047@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: mon.smithi047@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: mon.smithi047@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta expand map: {default=false} 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta from 'false' to 'false' 2024-04-22T22:16:28.077 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta expanded map: {default=false} 2024-04-22T22:16:28.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta expand map: {default=info} 2024-04-22T22:16:28.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta from 'info' to 'info' 2024-04-22T22:16:28.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta expanded map: {default=info} 2024-04-22T22:16:28.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta expand map: {default=daemon} 2024-04-22T22:16:28.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-22T22:16:28.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta expanded map: {default=daemon} 2024-04-22T22:16:28.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta expand map: {default=debug} 2024-04-22T22:16:28.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta from 'debug' to 'debug' 2024-04-22T22:16:28.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: expand_channel_meta expanded map: {default=debug} 2024-04-22T22:16:28.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 bash[21827]: 96cda5acb6353787a18bdf57e2fadf57f04f565897236605ed28e7993f67458e 2024-04-22T22:16:28.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 systemd[1]: Started Ceph mon.smithi047 for dbcf7660-00f5-11ef-bc93-c7b262605968. 2024-04-22T22:16:28.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: mon.smithi047 is new leader, mons smithi047 in quorum (ranks 0) 2024-04-22T22:16:28.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: monmap e1: 1 mons at {smithi047=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0]} 2024-04-22T22:16:28.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: fsmap 2024-04-22T22:16:28.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T22:16:28.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:28 smithi047 ceph-mon[21846]: mgrmap e1: no daemons active 2024-04-22T22:16:29.635 INFO:teuthology.orchestra.run.smithi047.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-22T22:16:29.635 INFO:teuthology.orchestra.run.smithi047.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:16:29.636 INFO:teuthology.orchestra.run.smithi047.stderr:Creating mgr... 2024-04-22T22:16:29.636 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying port 9283 ... 2024-04-22T22:16:29.822 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to reset failed state of unit ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mgr.smithi047.uutpnj.service: Unit ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mgr.smithi047.uutpnj.service not loaded. 2024-04-22T22:16:29.830 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dbcf7660-00f5-11ef-bc93-c7b262605968.target.wants/ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mgr.smithi047.uutpnj.service → /etc/systemd/system/ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@.service. 2024-04-22T22:16:30.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:29 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1732508253' entity='client.admin' 2024-04-22T22:16:30.515 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2024-04-22T22:16:30.522 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-04-22T22:16:30.522 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2024-04-22T22:16:30.523 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to enable service . firewalld.service is not available 2024-04-22T22:16:30.531 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2024-04-22T22:16:30.538 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-04-22T22:16:30.539 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2024-04-22T22:16:30.539 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-22T22:16:30.539 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr to start... 2024-04-22T22:16:30.539 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr... 2024-04-22T22:16:31.126 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:31.126 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-04-22T22:16:31.126 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsid": "dbcf7660-00f5-11ef-bc93-c7b262605968", 2024-04-22T22:16:31.126 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "health": { 2024-04-22T22:16:31.126 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T22:16:31.126 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T22:16:31.126 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 0 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "smithi047" 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_age": 3, 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T22:16:31.127 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T22:16:31.128 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:31.129 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T22:16:31.130 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": false, 2024-04-22T22:16:31.130 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T22:16:31.130 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T22:16:31.130 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "iostat", 2024-04-22T22:16:31.130 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "nfs", 2024-04-22T22:16:31.130 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "restful" 2024-04-22T22:16:31.130 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:31.130 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:16:31.130 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:31.130 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T22:16:31.130 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:31.131 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modified": "2024-04-22T22:16:23.412447+0000", 2024-04-22T22:16:31.131 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:16:31.131 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:31.131 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T22:16:31.131 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-04-22T22:16:31.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:31 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2661380382' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T22:16:31.632 INFO:teuthology.orchestra.run.smithi047.stderr:mgr not available, waiting (1/15)... 2024-04-22T22:16:34.310 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:34.310 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-04-22T22:16:34.310 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsid": "dbcf7660-00f5-11ef-bc93-c7b262605968", 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "health": { 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 0 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "smithi047" 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T22:16:34.311 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:34.312 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T22:16:34.312 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T22:16:34.312 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:34.312 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T22:16:34.312 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:34.316 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T22:16:34.316 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T22:16:34.317 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T22:16:34.317 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T22:16:34.320 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T22:16:34.320 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:34.321 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": false, 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "iostat", 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "nfs", 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "restful" 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:34.322 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modified": "2024-04-22T22:16:23.412447+0000", 2024-04-22T22:16:34.323 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:16:34.323 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:34.323 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T22:16:34.323 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-04-22T22:16:34.626 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:34 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2982608505' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T22:16:34.933 INFO:teuthology.orchestra.run.smithi047.stderr:mgr not available, waiting (2/15)... 2024-04-22T22:16:37.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: Activating manager daemon smithi047.uutpnj 2024-04-22T22:16:37.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: mgrmap e2: smithi047.uutpnj(active, starting, since 0.0029341s) 2024-04-22T22:16:37.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: from='mgr.14100 172.21.15.47:0/3955785167' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T22:16:37.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: from='mgr.14100 172.21.15.47:0/3955785167' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T22:16:37.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: from='mgr.14100 172.21.15.47:0/3955785167' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T22:16:37.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: from='mgr.14100 172.21.15.47:0/3955785167' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-04-22T22:16:37.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: from='mgr.14100 172.21.15.47:0/3955785167' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr metadata", "who": "smithi047.uutpnj", "id": "smithi047.uutpnj"}]: dispatch 2024-04-22T22:16:37.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: Manager daemon smithi047.uutpnj is now available 2024-04-22T22:16:37.059 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: from='mgr.14100 172.21.15.47:0/3955785167' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:16:37.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: from='mgr.14100 172.21.15.47:0/3955785167' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:16:37.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: from='mgr.14100 172.21.15.47:0/3955785167' entity='mgr.smithi047.uutpnj' 2024-04-22T22:16:37.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: from='mgr.14100 172.21.15.47:0/3955785167' entity='mgr.smithi047.uutpnj' 2024-04-22T22:16:37.060 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:36 smithi047 ceph-mon[21846]: from='mgr.14100 172.21.15.47:0/3955785167' entity='mgr.smithi047.uutpnj' 2024-04-22T22:16:37.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:37.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-04-22T22:16:37.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsid": "dbcf7660-00f5-11ef-bc93-c7b262605968", 2024-04-22T22:16:37.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "health": { 2024-04-22T22:16:37.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T22:16:37.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T22:16:37.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T22:16:37.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:37.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T22:16:37.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T22:16:37.423 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 0 2024-04-22T22:16:37.423 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:37.423 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T22:16:37.423 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "smithi047" 2024-04-22T22:16:37.423 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:37.423 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-04-22T22:16:37.423 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T22:16:37.423 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T22:16:37.424 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": false, 2024-04-22T22:16:37.425 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "iostat", 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "nfs", 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "restful" 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modified": "2024-04-22T22:16:23.412447+0000", 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T22:16:37.426 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-04-22T22:16:38.014 INFO:teuthology.orchestra.run.smithi047.stderr:mgr not available, waiting (3/15)... 2024-04-22T22:16:38.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:37 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/3529981345' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T22:16:38.067 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:37 smithi047 ceph-mon[21846]: mgrmap e3: smithi047.uutpnj(active, since 1.00628s) 2024-04-22T22:16:40.355 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:40 smithi047 ceph-mon[21846]: mgrmap e4: smithi047.uutpnj(active, since 2s) 2024-04-22T22:16:40.722 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:40.722 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-04-22T22:16:40.722 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsid": "dbcf7660-00f5-11ef-bc93-c7b262605968", 2024-04-22T22:16:40.722 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "health": { 2024-04-22T22:16:40.722 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T22:16:40.722 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T22:16:40.722 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T22:16:40.722 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:40.722 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 0 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "smithi047" 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T22:16:40.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T22:16:40.724 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T22:16:40.724 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T22:16:40.730 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T22:16:40.730 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T22:16:40.730 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:40.730 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T22:16:40.730 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T22:16:40.730 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T22:16:40.730 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": true, 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T22:16:40.731 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "iostat", 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "nfs", 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "restful" 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modified": "2024-04-22T22:16:23.412447+0000", 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T22:16:40.732 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-04-22T22:16:41.266 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:41 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/3217005428' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T22:16:41.269 INFO:teuthology.orchestra.run.smithi047.stderr:mgr is available 2024-04-22T22:16:41.898 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:41.898 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [global] 2024-04-22T22:16:41.898 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: fsid = dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:16:41.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-22T22:16:41.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-22T22:16:41.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-22T22:16:41.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-22T22:16:41.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:41.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [mgr] 2024-04-22T22:16:41.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-22T22:16:41.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-04-22T22:16:41.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [osd] 2024-04-22T22:16:41.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-22T22:16:41.899 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-22T22:16:42.259 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:42 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/984432800' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T22:16:42.527 INFO:teuthology.orchestra.run.smithi047.stderr:Enabling cephadm module... 2024-04-22T22:16:44.327 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:44 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1130206518' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T22:16:45.340 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:45 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1130206518' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T22:16:45.340 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:45 smithi047 ceph-mon[21846]: mgrmap e5: smithi047.uutpnj(active, since 7s) 2024-04-22T22:16:45.341 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-04-22T22:16:45.341 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-22T22:16:45.341 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": true, 2024-04-22T22:16:45.342 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "active_name": "smithi047.uutpnj", 2024-04-22T22:16:45.342 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-22T22:16:45.342 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-04-22T22:16:45.896 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for the mgr to restart... 2024-04-22T22:16:45.896 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr epoch 5... 2024-04-22T22:16:46.301 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:46 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2814544479' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T22:16:51.391 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-04-22T22:16:51.391 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-22T22:16:51.391 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "initialized": true 2024-04-22T22:16:51.391 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-04-22T22:16:51.627 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: Active manager daemon smithi047.uutpnj restarted 2024-04-22T22:16:51.627 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: Activating manager daemon smithi047.uutpnj 2024-04-22T22:16:51.627 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T22:16:51.627 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: mgrmap e6: smithi047.uutpnj(active, starting, since 0.0523261s) 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr metadata", "who": "smithi047.uutpnj", "id": "smithi047.uutpnj"}]: dispatch 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: Manager daemon smithi047.uutpnj is now available 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:16:51.628 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:51 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:16:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:mgr epoch 5 is available 2024-04-22T22:16:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:Setting orchestrator backend to cephadm... 2024-04-22T22:16:52.626 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:52 smithi047 ceph-mon[21846]: mgrmap e7: smithi047.uutpnj(active, since 1.05496s) 2024-04-22T22:16:52.627 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:52 smithi047 ceph-mon[21846]: [22/Apr/2024:22:16:51] ENGINE Bus STARTING 2024-04-22T22:16:52.627 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:52 smithi047 ceph-mon[21846]: [22/Apr/2024:22:16:51] ENGINE Serving on https://172.21.15.47:7150 2024-04-22T22:16:52.627 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:52 smithi047 ceph-mon[21846]: [22/Apr/2024:22:16:51] ENGINE Bus STARTED 2024-04-22T22:16:52.627 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:52 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:16:52.627 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:52 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:16:53.820 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:53 smithi047 ceph-mon[21846]: mgrmap e8: smithi047.uutpnj(active, since 2s) 2024-04-22T22:16:53.820 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:53 smithi047 ceph-mon[21846]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:16:53.820 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:53 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:16:53.820 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:53 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:16:53.821 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: value unchanged 2024-04-22T22:16:54.414 INFO:teuthology.orchestra.run.smithi047.stderr:Generating ssh key... 2024-04-22T22:16:54.730 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:54 smithi047 ceph-mon[21846]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:16:55.644 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:55 smithi047 ceph-mon[21846]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:16:55.644 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:55 smithi047 ceph-mon[21846]: Generating ssh key... 2024-04-22T22:16:55.644 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:55 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:16:55.644 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:55 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4JtYXwfldX2kwy3bB5e4uqfJJLPZmoGAPlzDqd8glizXoFjGlTkpYBHlYZzIfQHd/lf9c8sQuZp4B/qGi18JRUrdWJUS5XtEXEb8q5J1JmXDnMwWLNV85Lde4vmJtySr9rWPhX7qD0bsMsCRWdLh7SilxWggKZhMk7E67oQaI8Ngp3wH1y6VvyMAUKr5+szZf8fC++EzKzG0Dd1GMGzfHmvxzmpQWkZOzgNASwNCA3vNf6ee/8V8UODjSU+Rgt93KXBgs0+o6jV1TiuekwgYEDYRkD6JYIzCYzQPREceBb0e+sfA+E+2ncpS8buudORnKmSHLQuwqJ0R9TYZ9ld4qrwDP6VvCPEutefHDWIUfIO5VTdfyzYlM+RK7NmkHQt5+fv4hSQshcLIUF6SQ1oSw9v5WZ0H+eTm7yC69CPo6xVdXX0fpzPl3cqy0MDrvRp5gYIjmqaWR35RJDTqsSN0InInbZRWLQYjMZNCJfRKihC4CERrwrg6J39DLEnwGVHU= ceph-dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:16:56.850 INFO:teuthology.orchestra.run.smithi047.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T22:16:56.851 INFO:teuthology.orchestra.run.smithi047.stderr:Adding key to root@localhost authorized_keys... 2024-04-22T22:16:56.851 INFO:teuthology.orchestra.run.smithi047.stderr:Adding host smithi047... 2024-04-22T22:16:57.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:57 smithi047 ceph-mon[21846]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:16:58.340 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Added host 'smithi047' with addr '172.21.15.47' 2024-04-22T22:16:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying mon service with default placement... 2024-04-22T22:16:59.274 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:59 smithi047 ceph-mon[21846]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi047", "addr": "172.21.15.47", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:16:59.274 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:59 smithi047 ceph-mon[21846]: Deploying cephadm binary to smithi047 2024-04-22T22:16:59.274 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:59 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:16:59.274 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:16:59 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:16:59.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-22T22:17:00.126 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:00 smithi047 ceph-mon[21846]: Added host smithi047 2024-04-22T22:17:00.127 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:00 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:00.278 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying mgr service with default placement... 2024-04-22T22:17:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-22T22:17:01.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:01 smithi047 ceph-mon[21846]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:01.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:01 smithi047 ceph-mon[21846]: Saving service mon spec with placement count:5 2024-04-22T22:17:01.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:01 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:01.549 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying crash service with default placement... 2024-04-22T22:17:02.272 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:02 smithi047 ceph-mon[21846]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:02.273 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:02 smithi047 ceph-mon[21846]: Saving service mgr spec with placement count:2 2024-04-22T22:17:02.873 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled crash update... 2024-04-22T22:17:03.456 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying prometheus service with default placement... 2024-04-22T22:17:03.572 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:03 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:03.572 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:03 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:03.572 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:03 smithi047 ceph-mon[21846]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:03.572 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:03 smithi047 ceph-mon[21846]: Saving service crash spec with placement * 2024-04-22T22:17:03.572 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:03 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:04.071 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-04-22T22:17:05.116 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying grafana service with default placement... 2024-04-22T22:17:05.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:05 smithi047 ceph-mon[21846]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:05.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:05 smithi047 ceph-mon[21846]: Saving service prometheus spec with placement count:1 2024-04-22T22:17:05.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:05 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:05.719 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-04-22T22:17:06.356 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying node-exporter service with default placement... 2024-04-22T22:17:06.877 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:06 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:06.877 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:06 smithi047 ceph-mon[21846]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:06.877 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:06 smithi047 ceph-mon[21846]: Saving service grafana spec with placement count:1 2024-04-22T22:17:06.877 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:06 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:06.877 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:06 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:07.012 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-04-22T22:17:08.190 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:08 smithi047 ceph-mon[21846]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:08.190 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:08 smithi047 ceph-mon[21846]: Saving service node-exporter spec with placement * 2024-04-22T22:17:08.191 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:08 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:08.461 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying alertmanager service with default placement... 2024-04-22T22:17:09.121 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-04-22T22:17:10.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:10 smithi047 ceph-mon[21846]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:10.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:10 smithi047 ceph-mon[21846]: Saving service alertmanager spec with placement count:1 2024-04-22T22:17:10.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:10 smithi047 ceph-mon[21846]: from='mgr.14120 172.21.15.47:0/3204915159' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:11.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:11 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/389049519' entity='client.admin' 2024-04-22T22:17:11.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:11 smithi047 ceph-mon[21846]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:17:12.230 INFO:teuthology.orchestra.run.smithi047.stderr:Enabling the dashboard module... 2024-04-22T22:17:12.877 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:12 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/3273509458' entity='client.admin' 2024-04-22T22:17:13.942 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:13 smithi047 ceph-mon[21846]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:17:13.943 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:13 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/3780854982' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T22:17:14.877 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:14 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/3780854982' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T22:17:14.877 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:14 smithi047 ceph-mon[21846]: mgrmap e9: smithi047.uutpnj(active, since 23s) 2024-04-22T22:17:15.021 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-04-22T22:17:15.022 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-22T22:17:15.022 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": true, 2024-04-22T22:17:15.022 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "active_name": "smithi047.uutpnj", 2024-04-22T22:17:15.022 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-22T22:17:15.022 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-04-22T22:17:15.570 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for the mgr to restart... 2024-04-22T22:17:15.570 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr epoch 9... 2024-04-22T22:17:15.820 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:15 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/995246232' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T22:17:21.015 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-04-22T22:17:21.015 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-22T22:17:21.015 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "initialized": true 2024-04-22T22:17:21.015 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-04-22T22:17:21.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: Active manager daemon smithi047.uutpnj restarted 2024-04-22T22:17:21.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: Activating manager daemon smithi047.uutpnj 2024-04-22T22:17:21.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T22:17:21.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: mgrmap e10: smithi047.uutpnj(active, starting, since 0.0526959s) 2024-04-22T22:17:21.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-04-22T22:17:21.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr metadata", "who": "smithi047.uutpnj", "id": "smithi047.uutpnj"}]: dispatch 2024-04-22T22:17:21.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T22:17:21.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T22:17:21.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T22:17:21.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: Manager daemon smithi047.uutpnj is now available 2024-04-22T22:17:21.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:17:21.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:17:21.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:17:21.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:20 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:17:21.646 INFO:teuthology.orchestra.run.smithi047.stderr:mgr epoch 9 is available 2024-04-22T22:17:21.647 INFO:teuthology.orchestra.run.smithi047.stderr:Generating a dashboard self-signed certificate... 2024-04-22T22:17:22.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:22 smithi047 ceph-mon[21846]: [22/Apr/2024:22:17:20] ENGINE Bus STARTING 2024-04-22T22:17:22.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:22 smithi047 ceph-mon[21846]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T22:17:22.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:22 smithi047 ceph-mon[21846]: mgrmap e11: smithi047.uutpnj(active, since 1.05538s) 2024-04-22T22:17:22.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:22 smithi047 ceph-mon[21846]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T22:17:22.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:22 smithi047 ceph-mon[21846]: [22/Apr/2024:22:17:21] ENGINE Serving on https://172.21.15.47:7150 2024-04-22T22:17:22.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:22 smithi047 ceph-mon[21846]: [22/Apr/2024:22:17:21] ENGINE Bus STARTED 2024-04-22T22:17:22.261 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:22 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:22.261 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:22 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-22T22:17:23.045 INFO:teuthology.orchestra.run.smithi047.stderr:Creating initial admin user... 2024-04-22T22:17:23.437 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:23 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:23.437 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:23 smithi047 ceph-mon[21846]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:23.437 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:23 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:23.437 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:23 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:23.437 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:23 smithi047 ceph-mon[21846]: mgrmap e12: smithi047.uutpnj(active, since 2s) 2024-04-22T22:17:23.901 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$PLv8dGzSmhc.TFa4TtTyku77HzQI1tCk6CRC.eyAp1VhiyyTxQYba", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713824243, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T22:17:24.482 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:24 smithi047 ceph-mon[21846]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:24.482 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:24 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:24.802 INFO:teuthology.orchestra.run.smithi047.stderr:Fetching dashboard port number... 2024-04-22T22:17:25.843 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 8443 2024-04-22T22:17:26.128 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:25 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1012610625' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T22:17:26.498 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2024-04-22T22:17:26.506 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-04-22T22:17:26.507 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2024-04-22T22:17:26.507 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T22:17:26.509 INFO:teuthology.orchestra.run.smithi047.stderr:Ceph Dashboard is now available at: 2024-04-22T22:17:26.509 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-04-22T22:17:26.509 INFO:teuthology.orchestra.run.smithi047.stderr: URL: https://smithi047:8443/ 2024-04-22T22:17:26.509 INFO:teuthology.orchestra.run.smithi047.stderr: User: admin 2024-04-22T22:17:26.509 INFO:teuthology.orchestra.run.smithi047.stderr: Password: 9yb8ib7tcd 2024-04-22T22:17:26.509 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-04-22T22:17:26.509 INFO:teuthology.orchestra.run.smithi047.stderr:Enabling autotune for osd_memory_target 2024-04-22T22:17:28.109 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:27 smithi047 ceph-mon[21846]: mgrmap e13: smithi047.uutpnj(active, since 6s) 2024-04-22T22:17:28.513 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-22T22:17:29.099 INFO:teuthology.orchestra.run.smithi047.stderr:You can access the Ceph CLI with: 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr: ceph telemetry on 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr:For more information see: 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-04-22T22:17:29.100 INFO:teuthology.orchestra.run.smithi047.stderr:Bootstrap complete. 2024-04-22T22:17:29.137 INFO:tasks.cephadm:Fetching config... 2024-04-22T22:17:29.138 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-22T22:17:29.138 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T22:17:29.156 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T22:17:29.156 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-22T22:17:29.156 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T22:17:29.211 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T22:17:29.211 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-22T22:17:29.211 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/keyring of=/dev/stdout 2024-04-22T22:17:29.277 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T22:17:29.277 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-22T22:17:29.277 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T22:17:29.333 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T22:17:29.333 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4JtYXwfldX2kwy3bB5e4uqfJJLPZmoGAPlzDqd8glizXoFjGlTkpYBHlYZzIfQHd/lf9c8sQuZp4B/qGi18JRUrdWJUS5XtEXEb8q5J1JmXDnMwWLNV85Lde4vmJtySr9rWPhX7qD0bsMsCRWdLh7SilxWggKZhMk7E67oQaI8Ngp3wH1y6VvyMAUKr5+szZf8fC++EzKzG0Dd1GMGzfHmvxzmpQWkZOzgNASwNCA3vNf6ee/8V8UODjSU+Rgt93KXBgs0+o6jV1TiuekwgYEDYRkD6JYIzCYzQPREceBb0e+sfA+E+2ncpS8buudORnKmSHLQuwqJ0R9TYZ9ld4qrwDP6VvCPEutefHDWIUfIO5VTdfyzYlM+RK7NmkHQt5+fv4hSQshcLIUF6SQ1oSw9v5WZ0H+eTm7yC69CPo6xVdXX0fpzPl3cqy0MDrvRp5gYIjmqaWR35RJDTqsSN0InInbZRWLQYjMZNCJfRKihC4CERrwrg6J39DLEnwGVHU= ceph-dbcf7660-00f5-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T22:17:29.407 INFO:teuthology.orchestra.run.smithi047.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4JtYXwfldX2kwy3bB5e4uqfJJLPZmoGAPlzDqd8glizXoFjGlTkpYBHlYZzIfQHd/lf9c8sQuZp4B/qGi18JRUrdWJUS5XtEXEb8q5J1JmXDnMwWLNV85Lde4vmJtySr9rWPhX7qD0bsMsCRWdLh7SilxWggKZhMk7E67oQaI8Ngp3wH1y6VvyMAUKr5+szZf8fC++EzKzG0Dd1GMGzfHmvxzmpQWkZOzgNASwNCA3vNf6ee/8V8UODjSU+Rgt93KXBgs0+o6jV1TiuekwgYEDYRkD6JYIzCYzQPREceBb0e+sfA+E+2ncpS8buudORnKmSHLQuwqJ0R9TYZ9ld4qrwDP6VvCPEutefHDWIUfIO5VTdfyzYlM+RK7NmkHQt5+fv4hSQshcLIUF6SQ1oSw9v5WZ0H+eTm7yC69CPo6xVdXX0fpzPl3cqy0MDrvRp5gYIjmqaWR35RJDTqsSN0InInbZRWLQYjMZNCJfRKihC4CERrwrg6J39DLEnwGVHU= ceph-dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:17:29.418 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4JtYXwfldX2kwy3bB5e4uqfJJLPZmoGAPlzDqd8glizXoFjGlTkpYBHlYZzIfQHd/lf9c8sQuZp4B/qGi18JRUrdWJUS5XtEXEb8q5J1JmXDnMwWLNV85Lde4vmJtySr9rWPhX7qD0bsMsCRWdLh7SilxWggKZhMk7E67oQaI8Ngp3wH1y6VvyMAUKr5+szZf8fC++EzKzG0Dd1GMGzfHmvxzmpQWkZOzgNASwNCA3vNf6ee/8V8UODjSU+Rgt93KXBgs0+o6jV1TiuekwgYEDYRkD6JYIzCYzQPREceBb0e+sfA+E+2ncpS8buudORnKmSHLQuwqJ0R9TYZ9ld4qrwDP6VvCPEutefHDWIUfIO5VTdfyzYlM+RK7NmkHQt5+fv4hSQshcLIUF6SQ1oSw9v5WZ0H+eTm7yC69CPo6xVdXX0fpzPl3cqy0MDrvRp5gYIjmqaWR35RJDTqsSN0InInbZRWLQYjMZNCJfRKihC4CERrwrg6J39DLEnwGVHU= ceph-dbcf7660-00f5-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T22:17:29.458 INFO:teuthology.orchestra.run.smithi113.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4JtYXwfldX2kwy3bB5e4uqfJJLPZmoGAPlzDqd8glizXoFjGlTkpYBHlYZzIfQHd/lf9c8sQuZp4B/qGi18JRUrdWJUS5XtEXEb8q5J1JmXDnMwWLNV85Lde4vmJtySr9rWPhX7qD0bsMsCRWdLh7SilxWggKZhMk7E67oQaI8Ngp3wH1y6VvyMAUKr5+szZf8fC++EzKzG0Dd1GMGzfHmvxzmpQWkZOzgNASwNCA3vNf6ee/8V8UODjSU+Rgt93KXBgs0+o6jV1TiuekwgYEDYRkD6JYIzCYzQPREceBb0e+sfA+E+2ncpS8buudORnKmSHLQuwqJ0R9TYZ9ld4qrwDP6VvCPEutefHDWIUfIO5VTdfyzYlM+RK7NmkHQt5+fv4hSQshcLIUF6SQ1oSw9v5WZ0H+eTm7yC69CPo6xVdXX0fpzPl3cqy0MDrvRp5gYIjmqaWR35RJDTqsSN0InInbZRWLQYjMZNCJfRKihC4CERrwrg6J39DLEnwGVHU= ceph-dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:17:29.470 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T22:17:29.781 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:29 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1319067486' entity='client.admin' 2024-04-22T22:17:32.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:32 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:32.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:32 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd/host:smithi047", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:17:32.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:32 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:32.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:32 smithi047 ceph-mon[21846]: Deploying daemon alertmanager.smithi047 on smithi047 2024-04-22T22:17:32.986 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T22:17:32.987 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T22:17:33.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:33 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/3921974908' entity='client.admin' 2024-04-22T22:17:36.003 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi113 2024-04-22T22:17:36.003 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-22T22:17:36.003 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/etc/ceph/ceph.conf 2024-04-22T22:17:36.022 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-22T22:17:36.022 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:17:36.080 INFO:tasks.cephadm:Adding host smithi113 to orchestrator... 2024-04-22T22:17:36.081 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch host add smithi113 2024-04-22T22:17:36.753 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:36 smithi047 ceph-mon[21846]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:36.753 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:36 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:39.584 INFO:teuthology.orchestra.run.smithi047.stdout:Added host 'smithi113' with addr '172.21.15.113' 2024-04-22T22:17:40.255 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T22:17:40.878 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:40 smithi047 ceph-mon[21846]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi113", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:40.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:40 smithi047 ceph-mon[21846]: Deploying cephadm binary to smithi113 2024-04-22T22:17:40.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:40 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:40.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:40 smithi047 ceph-mon[21846]: Added host smithi113 2024-04-22T22:17:41.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:41 smithi047 ceph-mon[21846]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:17:43.103 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:17:43.103 INFO:teuthology.orchestra.run.smithi047.stdout:[{"addr": "172.21.15.47", "hostname": "smithi047", "labels": [], "status": ""}, {"addr": "172.21.15.113", "hostname": "smithi113", "labels": [], "status": ""}] 2024-04-22T22:17:43.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:43 smithi047 ceph-mon[21846]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:17:44.320 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T22:17:44.320 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T22:17:44.760 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:44 smithi047 ceph-mon[21846]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:17:45.869 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:45 smithi047 ceph-mon[21846]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:17:47.755 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:47 smithi047 ceph-mon[21846]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:17:48.598 INFO:teuthology.orchestra.run.smithi047.stderr:adjusted tunables profile to default 2024-04-22T22:17:48.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:48 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/998474589' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T22:17:48.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:48 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:48.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:48 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi047", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T22:17:48.880 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:48 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi047", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T22:17:48.880 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:48 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:17:49.810 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:49 smithi047 ceph-mon[21846]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:17:49.810 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:49 smithi047 ceph-mon[21846]: Deploying daemon crash.smithi047 on smithi047 2024-04-22T22:17:49.810 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:49 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/998474589' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T22:17:49.810 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:49 smithi047 ceph-mon[21846]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:17:49.811 INFO:tasks.cephadm:Adding mon.smithi047 on smithi047 2024-04-22T22:17:49.812 INFO:tasks.cephadm:Adding mon.smithi113 on smithi113 2024-04-22T22:17:49.812 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch apply mon '2;smithi047:172.21.15.47=smithi047;smithi113:172.21.15.113=smithi113' 2024-04-22T22:17:51.379 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:51 smithi047 ceph-mon[21846]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:17:51.379 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:51 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:51.560 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled mon update... 2024-04-22T22:17:52.097 DEBUG:teuthology.orchestra.run.smithi113:mon.smithi113> sudo journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mon.smithi113.service 2024-04-22T22:17:52.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:17:52.099 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:17:52.379 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:52 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:52.379 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:52 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:52.379 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:52 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:52.379 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:52 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T22:17:52.379 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:52 smithi047 ceph-mon[21846]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T22:17:52.379 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:52 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:52.380 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:52 smithi047 ceph-mon[21846]: Deploying daemon grafana.smithi047 on smithi047 2024-04-22T22:17:52.380 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:52 smithi047 ceph-mon[21846]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi047:172.21.15.47=smithi047;smithi113:172.21.15.113=smithi113", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:17:52.380 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:52 smithi047 ceph-mon[21846]: Saving service mon spec with placement smithi047:172.21.15.47=smithi047;smithi113:172.21.15.113=smithi113;count:2 2024-04-22T22:17:52.380 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:52 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:53.879 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:17:53.879 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:16:20.165981Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:17:53.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:53 smithi047 ceph-mon[21846]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:17:53.880 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-04-22T22:17:54.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:54 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/795183326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:17:55.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:17:55.482 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:17:55.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:55 smithi047 ceph-mon[21846]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:17:55.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:55 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:17:57.341 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:17:57.342 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:16:20.165981Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:17:57.343 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-04-22T22:17:57.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:57 smithi047 ceph-mon[21846]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:17:57.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:57 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/2963981852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:17:58.846 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:17:58.846 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:17:59.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:17:59 smithi047 ceph-mon[21846]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:00.671 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:18:00.671 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:16:20.165981Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:18:00.672 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-04-22T22:18:01.846 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:01 smithi047 ceph-mon[21846]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:01.846 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:01 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/2149443374' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:02.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:18:02.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:18:03.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:03 smithi047 ceph-mon[21846]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:04.121 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:18:04.122 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:16:20.165981Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:18:04.123 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-04-22T22:18:04.878 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:04 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/2137041690' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:05.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:18:05.722 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:18:05.880 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:05 smithi047 ceph-mon[21846]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:05.880 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:05 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:05.880 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:05 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:06.630 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:06 smithi047 ceph-mon[21846]: Deploying daemon node-exporter.smithi047 on smithi047 2024-04-22T22:18:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:18:07.444 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:16:20.165981Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:18:07.446 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-04-22T22:18:07.755 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:07 smithi047 ceph-mon[21846]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:07.755 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:07 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/374353572' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:09.008 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:18:09.008 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:18:09.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:08 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:09.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:08 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T22:18:09.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:08 smithi047 ceph-mon[21846]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:10.383 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:09 smithi047 ceph-mon[21846]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T22:18:10.383 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:09 smithi047 ceph-mon[21846]: mgrmap e14: smithi047.uutpnj(active, since 48s) 2024-04-22T22:18:10.709 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:18:10.709 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:16:20.165981Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:18:10.710 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-04-22T22:18:11.158 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:10 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/556399557' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:12.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:18:12.297 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:18:14.040 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:18:14.040 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:16:20.165981Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:18:14.041 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-04-22T22:18:14.380 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:14 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/3837724631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:15.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:18:15.498 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:18:16.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: Active manager daemon smithi047.uutpnj restarted 2024-04-22T22:18:16.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: Activating manager daemon smithi047.uutpnj 2024-04-22T22:18:16.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T22:18:16.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: mgrmap e15: smithi047.uutpnj(active, starting, since 0.0528547s) 2024-04-22T22:18:16.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-04-22T22:18:16.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr metadata", "who": "smithi047.uutpnj", "id": "smithi047.uutpnj"}]: dispatch 2024-04-22T22:18:16.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T22:18:16.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T22:18:16.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T22:18:16.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: Manager daemon smithi047.uutpnj is now available 2024-04-22T22:18:16.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:18:16.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:18:16.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:16.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:18:16.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:16.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:18:16.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:18:16.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: [22/Apr/2024:22:18:15] ENGINE Bus STARTING 2024-04-22T22:18:16.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:16.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: [22/Apr/2024:22:18:15] ENGINE Serving on https://172.21.15.47:7150 2024-04-22T22:18:16.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: [22/Apr/2024:22:18:15] ENGINE Bus STARTED 2024-04-22T22:18:16.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:17.512 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:18:17.513 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:16:20.165981Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:18:17.515 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-04-22T22:18:17.630 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:17 smithi047 ceph-mon[21846]: mgrmap e16: smithi047.uutpnj(active, since 1.05528s) 2024-04-22T22:18:17.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:17 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:18.880 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:18 smithi047 ceph-mon[21846]: mgrmap e17: smithi047.uutpnj(active, since 2s) 2024-04-22T22:18:18.880 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:18 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/279737075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:18.880 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:18 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:19.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:18:19.114 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:18:20.130 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:20.131 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:20.131 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd/host:smithi047", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:18:20.131 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:21.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:20 smithi047 ceph-mon[21846]: Updating smithi047:/etc/ceph/ceph.conf 2024-04-22T22:18:21.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:20 smithi047 ceph-mon[21846]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:18:21.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:21.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:21.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd/host:smithi113", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:18:21.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:21.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T22:18:21.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T22:18:21.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:21.958 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:18:21.958 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:16:20.165981Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:18:21.960 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-04-22T22:18:22.380 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:21 smithi047 ceph-mon[21846]: Deploying daemon crash.smithi113 on smithi113 2024-04-22T22:18:23.380 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:22 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/2077557333' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:24.265 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:18:24.266 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:18:25.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:25.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.qvxssq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:18:25.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.qvxssq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T22:18:25.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:18:25.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:25.381 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:25 smithi047 ceph-mon[21846]: Deploying daemon mgr.smithi113.qvxssq on smithi113 2024-04-22T22:18:26.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:26 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:26.918 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:18:26.918 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:16:20.165981Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:18:26.920 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-04-22T22:18:27.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:27 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/2959746628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:28.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:28.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:18:28.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:28.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:28 smithi047 ceph-mon[21846]: Deploying daemon mon.smithi113 on smithi113 2024-04-22T22:18:28.742 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:18:28.743 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:18:31.167 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:18:31.167 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:16:20.165981Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:18:31.169 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-04-22T22:18:31.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:31 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:31.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:31 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/1978364726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:31.730 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:31 smithi113 systemd[1]: Starting Ceph mon.smithi113 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:18:32.350 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 podman[24062]: 2024-04-22 22:18:32.010832765 +0000 UTC m=+0.018250674 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:18:32.350 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 podman[24062]: 2024-04-22 22:18:32.111692021 +0000 UTC m=+0.119109930 container create 82b94d1b7aafd88134458827c0a91bb026cde160e5f0c5065e781db4635eb37c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi113, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.tags=base centos centos-stream, name=centos-stream, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, release=754, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vendor=Red Hat, Inc., architecture=x86_64, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True) 2024-04-22T22:18:32.350 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 podman[24062]: 2024-04-22 22:18:32.372923146 +0000 UTC m=+0.380341056 container init 82b94d1b7aafd88134458827c0a91bb026cde160e5f0c5065e781db4635eb37c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi113, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, ceph=True, vcs-type=git, distribution-scope=public, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , release=754, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T22:18:32.735 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 podman[24062]: 2024-04-22 22:18:32.376960883 +0000 UTC m=+0.384378793 container start 82b94d1b7aafd88134458827c0a91bb026cde160e5f0c5065e781db4635eb37c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi113, distribution-scope=public, release=754, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , ceph=True, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vendor=Red Hat, Inc., version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pidfile_write: ignore empty --pid-file 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: load: jerasure load: lrc 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: RocksDB version: 6.15.5 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Compile date Apr 18 2022 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: DB SUMMARY 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: DB Session ID: 03G00L5XL10TW6AD5YHT 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: CURRENT file: CURRENT 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi113/store.db dir, Total Num: 0, files: 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi113/store.db: 000004.log size: 511 ; 2024-04-22T22:18:32.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.error_if_exists: 0 2024-04-22T22:18:32.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.create_if_missing: 0 2024-04-22T22:18:32.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.paranoid_checks: 1 2024-04-22T22:18:32.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T22:18:32.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.env: 0x55720c8c6860 2024-04-22T22:18:32.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.fs: Posix File System 2024-04-22T22:18:32.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.info_log: 0x55720dba3fa0 2024-04-22T22:18:32.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.statistics: (nil) 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.use_fsync: 0 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_log_file_size: 0 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.allow_fallocate: 1 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.use_direct_reads: 0 2024-04-22T22:18:32.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.db_log_dir: 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi113/store.db 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.write_buffer_manager: 0x55720dc94330 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-22T22:18:32.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.unordered_write: 0 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.row_cache: None 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.wal_filter: None 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.preserve_deletes: 0 2024-04-22T22:18:32.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.two_write_queues: 0 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.atomic_flush: 0 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.log_readahead_size: 0 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_background_jobs: 2 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_background_compactions: -1 2024-04-22T22:18:32.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_subcompactions: 1 2024-04-22T22:18:32.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T22:18:32.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T22:18:32.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T22:18:32.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T22:18:32.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T22:18:32.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T22:18:32.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T22:18:32.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T22:18:32.745 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_open_files: -1 2024-04-22T22:18:32.745 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T22:18:32.745 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T22:18:32.745 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T22:18:32.745 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T22:18:32.745 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_background_flushes: -1 2024-04-22T22:18:32.745 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Compression algorithms supported: 2024-04-22T22:18:32.745 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T22:18:32.745 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: kZSTD supported: 0 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: kXpressCompression supported: 0 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: kLZ4Compression supported: 1 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: kBZip2Compression supported: 0 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: kZlibCompression supported: 1 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: kSnappyCompression supported: 1 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi113/store.db/MANIFEST-000003 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.merge_operator: 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_filter: None 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T22:18:32.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55720db6fd00) 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: cache_index_and_filter_blocks: 1 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: pin_top_level_index_and_filter: 1 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: index_type: 0 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: data_block_index_type: 0 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: index_shortening: 1 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: hash_index_allow_collision: 1 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: checksum: 1 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: no_block_cache: 0 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_cache: 0x55720dbda170 2024-04-22T22:18:32.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_cache_name: BinnedLRUCache 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_cache_options: 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: capacity : 536870912 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: num_shard_bits : 4 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: strict_capacity_limit : 0 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: high_pri_pool_ratio: 0.000 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_cache_compressed: (nil) 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: persistent_cache: (nil) 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_size: 4096 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_size_deviation: 10 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_restart_interval: 16 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: index_block_restart_interval: 1 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: metadata_block_size: 4096 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: partition_filters: 0 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: use_delta_encoding: 1 2024-04-22T22:18:32.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-22T22:18:32.749 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: whole_key_filtering: 1 2024-04-22T22:18:32.749 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: verify_compression: 0 2024-04-22T22:18:32.749 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: read_amp_bytes_per_bit: 0 2024-04-22T22:18:32.749 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: format_version: 4 2024-04-22T22:18:32.749 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: enable_index_compression: 1 2024-04-22T22:18:32.749 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_align: 0 2024-04-22T22:18:32.749 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T22:18:32.749 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T22:18:32.751 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compression: NoCompression 2024-04-22T22:18:32.751 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T22:18:32.751 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T22:18:32.751 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T22:18:32.751 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.num_levels: 7 2024-04-22T22:18:32.751 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T22:18:32.751 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T22:18:32.751 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T22:18:32.751 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T22:18:32.751 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T22:18:32.751 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T22:18:32.755 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T22:18:32.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.arena_block_size: 4194304 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T22:18:32.757 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.table_properties_collectors: 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.inplace_update_support: 0 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.bloom_locality: 0 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.max_successive_merges: 0 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.ttl: 2592000 2024-04-22T22:18:32.758 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.enable_blob_files: false 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.min_blob_size: 0 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi113/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713824312402689, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713824312402991, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713824312, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "ccfc75bf-2ec2-4413-8244-4ac906ecad64", "db_session_id": "03G00L5XL10TW6AD5YHT"}} 2024-04-22T22:18:32.759 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713824312403323, "job": 1, "event": "recovery_finished"} 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi113/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55720dbc0a80 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: DB pointer 0x55720dbd0000 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ** DB Stats ** 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-22T22:18:32.760 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ** Compaction Stats [default] ** 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ** Compaction Stats [default] ** 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T22:18:32.761 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Interval compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ** Compaction Stats [default] ** 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-22T22:18:32.762 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ** Compaction Stats [default] ** 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T22:18:32.763 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113 does not exist in monmap, will attempt to join an existing cluster 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: using public_addr v2:172.21.15.113:0/0 -> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: starting mon.smithi113 rank -1 at public addrs [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] at bind addrs [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi113 fsid dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(???) e0 preinit fsid dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).mds e1 new map 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).mds e1 print_map 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: e1 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: legacy client fscid: -1 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: No filesystems configured 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T22:18:32.764 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/998474589' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi047", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi047", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:32.765 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Deploying daemon crash.smithi047 on smithi047 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/998474589' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Deploying daemon grafana.smithi047 on smithi047 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi047:172.21.15.47=smithi047;smithi113:172.21.15.113=smithi113", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Saving service mon spec with placement smithi047:172.21.15.47=smithi047;smithi113:172.21.15.113=smithi113;count:2 2024-04-22T22:18:32.766 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/795183326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/2963981852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/2149443374' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/2137041690' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.767 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Deploying daemon node-exporter.smithi047 on smithi047 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/374353572' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14162 172.21.15.47:0/760212900' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mgrmap e14: smithi047.uutpnj(active, since 48s) 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/556399557' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/3837724631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Active manager daemon smithi047.uutpnj restarted 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Activating manager daemon smithi047.uutpnj 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mgrmap e15: smithi047.uutpnj(active, starting, since 0.0528547s) 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-04-22T22:18:32.768 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr metadata", "who": "smithi047.uutpnj", "id": "smithi047.uutpnj"}]: dispatch 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Manager daemon smithi047.uutpnj is now available 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: [22/Apr/2024:22:18:15] ENGINE Bus STARTING 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: [22/Apr/2024:22:18:15] ENGINE Serving on https://172.21.15.47:7150 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: [22/Apr/2024:22:18:15] ENGINE Bus STARTED 2024-04-22T22:18:32.769 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mgrmap e16: smithi047.uutpnj(active, since 1.05528s) 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mgrmap e17: smithi047.uutpnj(active, since 2s) 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/279737075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd/host:smithi047", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Updating smithi047:/etc/ceph/ceph.conf 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.770 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd/host:smithi113", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Deploying daemon crash.smithi113 on smithi113 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/2077557333' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.qvxssq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.qvxssq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Deploying daemon mgr.smithi113.qvxssq on smithi113 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/2959746628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.771 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.772 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:18:32.772 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:32.772 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: Deploying daemon mon.smithi113 on smithi113 2024-04-22T22:18:32.772 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:32.772 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/1978364726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:32.772 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-04-22T22:18:32.772 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 ceph-mon[24083]: mon.smithi113@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-22T22:18:32.772 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 bash[24062]: 82b94d1b7aafd88134458827c0a91bb026cde160e5f0c5065e781db4635eb37c 2024-04-22T22:18:32.772 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:32 smithi113 systemd[1]: Started Ceph mon.smithi113 for dbcf7660-00f5-11ef-bc93-c7b262605968. 2024-04-22T22:18:32.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T22:18:32.858 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:18:37.725 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-04-22T22:18:37.725 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: mon.smithi047 calling monitor election 2024-04-22T22:18:37.725 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.726 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.? 172.21.15.113:0/833325303' entity='mgr.smithi113.qvxssq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.qvxssq/crt"}]: dispatch 2024-04-22T22:18:37.726 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.726 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.726 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: mon.smithi113 calling monitor election 2024-04-22T22:18:37.726 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:37.726 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.726 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.726 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.726 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: mon.smithi047 is new leader, mons smithi047,smithi113 in quorum (ranks 0,1) 2024-04-22T22:18:37.726 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: monmap e2: 2 mons at {smithi047=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],smithi113=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} 2024-04-22T22:18:37.726 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: fsmap 2024-04-22T22:18:37.727 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T22:18:37.727 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: mgrmap e17: smithi047.uutpnj(active, since 22s) 2024-04-22T22:18:37.727 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: Standby manager daemon smithi113.qvxssq started 2024-04-22T22:18:37.727 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.? 172.21.15.113:0/833325303' entity='mgr.smithi113.qvxssq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T22:18:37.727 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.? 172.21.15.113:0/833325303' entity='mgr.smithi113.qvxssq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.qvxssq/key"}]: dispatch 2024-04-22T22:18:37.727 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.? 172.21.15.113:0/833325303' entity='mgr.smithi113.qvxssq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T22:18:37.727 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: overall HEALTH_OK 2024-04-22T22:18:37.727 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:37.853 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-22T22:18:37.853 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":2,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","modified":"2024-04-22T22:18:32.433627Z","created":"2024-04-22T22:16:20.165981Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi113","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T22:18:37.854 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 2 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: mon.smithi047 calling monitor election 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.? 172.21.15.113:0/833325303' entity='mgr.smithi113.qvxssq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.qvxssq/crt"}]: dispatch 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: mon.smithi113 calling monitor election 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:37.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: mon.smithi047 is new leader, mons smithi047,smithi113 in quorum (ranks 0,1) 2024-04-22T22:18:37.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: monmap e2: 2 mons at {smithi047=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],smithi113=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} 2024-04-22T22:18:37.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: fsmap 2024-04-22T22:18:37.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T22:18:37.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: mgrmap e17: smithi047.uutpnj(active, since 22s) 2024-04-22T22:18:37.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: Standby manager daemon smithi113.qvxssq started 2024-04-22T22:18:37.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.? 172.21.15.113:0/833325303' entity='mgr.smithi113.qvxssq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T22:18:37.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.? 172.21.15.113:0/833325303' entity='mgr.smithi113.qvxssq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.qvxssq/key"}]: dispatch 2024-04-22T22:18:37.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.? 172.21.15.113:0/833325303' entity='mgr.smithi113.qvxssq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T22:18:37.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: overall HEALTH_OK 2024-04-22T22:18:37.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:38.456 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T22:18:38.457 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T22:18:38.735 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:38 smithi113 ceph-mon[24083]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:38.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:38 smithi113 ceph-mon[24083]: Deploying daemon node-exporter.smithi113 on smithi113 2024-04-22T22:18:38.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:38 smithi113 ceph-mon[24083]: mgrmap e18: smithi047.uutpnj(active, since 22s), standbys: smithi113.qvxssq 2024-04-22T22:18:38.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:38 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr metadata", "who": "smithi113.qvxssq", "id": "smithi113.qvxssq"}]: dispatch 2024-04-22T22:18:38.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:38 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/2449379677' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:38.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:38 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:38.765 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:38 smithi047 ceph-mon[21846]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:38.765 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:38 smithi047 ceph-mon[21846]: Deploying daemon node-exporter.smithi113 on smithi113 2024-04-22T22:18:38.765 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:38 smithi047 ceph-mon[21846]: mgrmap e18: smithi047.uutpnj(active, since 22s), standbys: smithi113.qvxssq 2024-04-22T22:18:38.765 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:38 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr metadata", "who": "smithi113.qvxssq", "id": "smithi113.qvxssq"}]: dispatch 2024-04-22T22:18:38.766 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:38 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/2449379677' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:18:38.766 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:38 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:18:40.395 INFO:teuthology.orchestra.run.smithi047.stdout:# minimal ceph.conf for dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:18:40.395 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-04-22T22:18:40.395 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = dbcf7660-00f5-11ef-bc93-c7b262605968 2024-04-22T22:18:40.395 INFO:teuthology.orchestra.run.smithi047.stdout: mon_host = [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] 2024-04-22T22:18:40.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:40 smithi113 ceph-mon[24083]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:40.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:40 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:40.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:40 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/1425714423' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:40.881 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:40 smithi047 ceph-mon[21846]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:40.881 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:40 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:40.882 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:40 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1425714423' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:41.004 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T22:18:41.005 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-22T22:18:41.005 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T22:18:41.034 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-22T22:18:41.034 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:18:41.101 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-22T22:18:41.101 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T22:18:41.128 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-22T22:18:41.128 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:18:41.193 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T22:18:41.194 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-22T22:18:41.194 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T22:18:41.209 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T22:18:41.210 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_1 2024-04-22T22:18:41.267 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T22:18:41.267 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:18:41.267 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-22T22:18:41.267 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:18:41.267 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:18:41.267 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-04-22 22:18:05.335973319 +0000 2024-04-22T22:18:41.267 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-04-22 22:15:22.554833098 +0000 2024-04-22T22:18:41.267 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-04-22 22:15:22.554833098 +0000 2024-04-22T22:18:41.267 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-04-22 22:15:22.554833098 +0000 2024-04-22T22:18:41.268 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T22:18:41.330 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-04-22T22:18:41.330 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-04-22T22:18:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000114236 s, 4.5 MB/s 2024-04-22T22:18:41.332 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T22:18:41.388 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_2 2024-04-22T22:18:41.445 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T22:18:41.445 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:18:41.445 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T22:18:41.445 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:18:41.445 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:18:41.445 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-04-22 22:18:05.336973297 +0000 2024-04-22T22:18:41.446 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-04-22 22:15:22.909825355 +0000 2024-04-22T22:18:41.446 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-04-22 22:15:22.909825355 +0000 2024-04-22T22:18:41.446 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-04-22 22:15:22.909825355 +0000 2024-04-22T22:18:41.446 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T22:18:41.510 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-04-22T22:18:41.510 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-04-22T22:18:41.510 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000134042 s, 3.8 MB/s 2024-04-22T22:18:41.511 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T22:18:41.571 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_3 2024-04-22T22:18:41.626 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T22:18:41.627 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:18:41.627 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T22:18:41.627 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:18:41.627 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:18:41.627 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-04-22 22:18:05.336973297 +0000 2024-04-22T22:18:41.627 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-04-22 22:15:23.252817873 +0000 2024-04-22T22:18:41.627 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-04-22 22:15:23.252817873 +0000 2024-04-22T22:18:41.627 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-04-22 22:15:23.252817873 +0000 2024-04-22T22:18:41.627 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T22:18:41.693 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-04-22T22:18:41.693 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-04-22T22:18:41.693 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 9.7134e-05 s, 5.3 MB/s 2024-04-22T22:18:41.694 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T22:18:41.752 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_4 2024-04-22T22:18:41.809 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T22:18:41.810 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:18:41.810 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-22T22:18:41.810 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:18:41.810 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:18:41.810 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-04-22 22:18:05.336973297 +0000 2024-04-22T22:18:41.810 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-04-22 22:15:23.601810261 +0000 2024-04-22T22:18:41.810 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-04-22 22:15:23.601810261 +0000 2024-04-22T22:18:41.810 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-04-22 22:15:23.601810261 +0000 2024-04-22T22:18:41.811 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T22:18:41.875 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-04-22T22:18:41.875 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-04-22T22:18:41.875 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000102471 s, 5.0 MB/s 2024-04-22T22:18:41.876 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T22:18:41.932 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-22T22:18:41.932 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T22:18:41.948 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T22:18:41.948 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_1 2024-04-22T22:18:42.003 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T22:18:42.003 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:18:42.003 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-22T22:18:42.003 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:18:42.003 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:18:42.003 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-22 22:18:38.041841125 +0000 2024-04-22T22:18:42.003 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-22 22:15:20.270655582 +0000 2024-04-22T22:18:42.004 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-22 22:15:20.270655582 +0000 2024-04-22T22:18:42.004 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-04-22 22:15:20.270655582 +0000 2024-04-22T22:18:42.004 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T22:18:42.066 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-22T22:18:42.067 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-22T22:18:42.067 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000228568 s, 2.2 MB/s 2024-04-22T22:18:42.068 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T22:18:42.124 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_2 2024-04-22T22:18:42.179 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T22:18:42.179 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:18:42.179 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T22:18:42.179 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:18:42.179 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:18:42.179 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-22 22:18:38.041841125 +0000 2024-04-22T22:18:42.179 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-22 22:15:20.625648243 +0000 2024-04-22T22:18:42.179 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-22 22:15:20.625648243 +0000 2024-04-22T22:18:42.179 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-04-22 22:15:20.625648243 +0000 2024-04-22T22:18:42.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T22:18:42.241 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-22T22:18:42.241 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-22T22:18:42.241 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000193149 s, 2.7 MB/s 2024-04-22T22:18:42.242 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T22:18:42.297 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_3 2024-04-22T22:18:42.352 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T22:18:42.352 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:18:42.352 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T22:18:42.352 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:18:42.352 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:18:42.352 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-22 22:18:38.042841105 +0000 2024-04-22T22:18:42.352 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-22 22:15:21.022640036 +0000 2024-04-22T22:18:42.352 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-22 22:15:21.022640036 +0000 2024-04-22T22:18:42.352 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-04-22 22:15:21.022640036 +0000 2024-04-22T22:18:42.353 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T22:18:42.414 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-22T22:18:42.414 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-22T22:18:42.414 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000243554 s, 2.1 MB/s 2024-04-22T22:18:42.415 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T22:18:42.472 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_4 2024-04-22T22:18:42.526 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T22:18:42.527 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:18:42.527 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-22T22:18:42.527 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:18:42.527 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:18:42.527 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-22 22:18:38.042841105 +0000 2024-04-22T22:18:42.527 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-22 22:15:21.364632966 +0000 2024-04-22T22:18:42.527 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-22 22:15:21.364632966 +0000 2024-04-22T22:18:42.527 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-04-22 22:15:21.364632966 +0000 2024-04-22T22:18:42.527 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T22:18:42.589 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-22T22:18:42.589 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-22T22:18:42.589 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000255202 s, 2.0 MB/s 2024-04-22T22:18:42.590 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T22:18:42.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:42 smithi047 ceph-mon[21846]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:42.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:42 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:42.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:42 smithi047 ceph-mon[21846]: Deploying daemon prometheus.smithi047 on smithi047 2024-04-22T22:18:42.646 INFO:tasks.cephadm:Deploying osd.0 on smithi047 with /dev/vg_nvme/lv_4... 2024-04-22T22:18:42.646 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T22:18:42.735 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:42 smithi113 ceph-mon[24083]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:42.735 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:42 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:42.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:42 smithi113 ceph-mon[24083]: Deploying daemon prometheus.smithi047 on smithi047 2024-04-22T22:18:44.324 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:44 smithi047 ceph-mon[21846]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:44.485 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:44 smithi113 ceph-mon[24083]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:45.691 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T22:18:45.691 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T22:18:45.691 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2024-04-22T22:18:45.691 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2024-04-22T22:18:45.691 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00851451 s, 1.2 GB/s 2024-04-22T22:18:45.691 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2024-04-22T22:18:46.382 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:46 smithi047 ceph-mon[21846]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:46.382 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:46 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:46.735 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:46 smithi113 ceph-mon[24083]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:46.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:46 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:47.119 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi047:vg_nvme/lv_4 2024-04-22T22:18:48.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:48 smithi047 ceph-mon[21846]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:48.735 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:48 smithi113 ceph-mon[24083]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:50.525 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:50 smithi047 ceph-mon[21846]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:50.735 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:50 smithi113 ceph-mon[24083]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:51.607 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:51 smithi047 ceph-mon[21846]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:18:51.607 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:51 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:18:51.607 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:51 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:18:51.607 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:51 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:51.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:51 smithi113 ceph-mon[24083]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:18:51.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:51 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:18:51.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:51 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:18:51.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:51 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:52.570 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:52 smithi047 ceph-mon[21846]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:52.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:52 smithi113 ceph-mon[24083]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:54.132 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:54 smithi047 ceph-mon[21846]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:54.486 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:54 smithi113 ceph-mon[24083]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:56.132 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:56 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:56.132 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:56 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:18:56.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:56 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:56.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:56 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:18:56.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:56 smithi047 ceph-mon[21846]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:56.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:56 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:56.295 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:56 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:56.296 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:56 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:18:56.296 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:56 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:18:56.296 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:56 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:18:56.296 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:56 smithi113 ceph-mon[24083]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:56.296 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:56 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:57.132 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:57 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2260758192' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eaaaa67b-8d25-4a21-ad4f-299f01bbaa73"}]: dispatch 2024-04-22T22:18:57.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:57 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2260758192' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eaaaa67b-8d25-4a21-ad4f-299f01bbaa73"}]': finished 2024-04-22T22:18:57.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:57 smithi047 ceph-mon[21846]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T22:18:57.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:57 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:18:57.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:57 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1548346732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:18:57.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:57 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:57.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:57 smithi047 ceph-mon[21846]: Updating smithi113:/etc/ceph/ceph.conf 2024-04-22T22:18:57.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:57 smithi047 ceph-mon[21846]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:18:57.236 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:57 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2260758192' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eaaaa67b-8d25-4a21-ad4f-299f01bbaa73"}]: dispatch 2024-04-22T22:18:57.236 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:57 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2260758192' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eaaaa67b-8d25-4a21-ad4f-299f01bbaa73"}]': finished 2024-04-22T22:18:57.236 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:57 smithi113 ceph-mon[24083]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T22:18:57.236 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:57 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:18:57.236 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:57 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/1548346732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:18:57.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:57 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:57.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:57 smithi113 ceph-mon[24083]: Updating smithi113:/etc/ceph/ceph.conf 2024-04-22T22:18:57.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:57 smithi113 ceph-mon[24083]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:18:58.382 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:58.386 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:18:58 smithi047 ceph-mon[21846]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:18:58.486 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:18:58.486 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:18:58 smithi113 ceph-mon[24083]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:00.486 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:00.487 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:00 smithi113 ceph-mon[24083]: Updating smithi047:/etc/ceph/ceph.conf 2024-04-22T22:19:00.487 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:00 smithi113 ceph-mon[24083]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:00.487 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:00.487 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:00.487 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:00 smithi113 ceph-mon[24083]: Reconfiguring mon.smithi047 (unknown last config time)... 2024-04-22T22:19:00.487 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:19:00.487 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:19:00.487 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:00.487 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:00 smithi113 ceph-mon[24083]: Reconfiguring daemon mon.smithi047 on smithi047 2024-04-22T22:19:00.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:00.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:00 smithi047 ceph-mon[21846]: Updating smithi047:/etc/ceph/ceph.conf 2024-04-22T22:19:00.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:00 smithi047 ceph-mon[21846]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:00.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:00.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:00.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:00 smithi047 ceph-mon[21846]: Reconfiguring mon.smithi047 (unknown last config time)... 2024-04-22T22:19:00.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:19:00.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:19:00.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:00.631 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:00 smithi047 ceph-mon[21846]: Reconfiguring daemon mon.smithi047 on smithi047 2024-04-22T22:19:02.096 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:02.096 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:02 smithi047 ceph-mon[21846]: Reconfiguring mgr.smithi047.uutpnj (unknown last config time)... 2024-04-22T22:19:02.096 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi047.uutpnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:19:02.096 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:19:02.096 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:02.097 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:02 smithi047 ceph-mon[21846]: Reconfiguring daemon mgr.smithi047.uutpnj on smithi047 2024-04-22T22:19:02.097 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:02 smithi047 ceph-mon[21846]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:02.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:02.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:02 smithi113 ceph-mon[24083]: Reconfiguring mgr.smithi047.uutpnj (unknown last config time)... 2024-04-22T22:19:02.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi047.uutpnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:19:02.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:19:02.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:02.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:02 smithi113 ceph-mon[24083]: Reconfiguring daemon mgr.smithi047.uutpnj on smithi047 2024-04-22T22:19:02.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:02 smithi113 ceph-mon[24083]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:04.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:04 smithi047 ceph-mon[21846]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:04.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:04 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:04.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:04 smithi047 ceph-mon[21846]: Reconfiguring alertmanager.smithi047 (dependencies changed)... 2024-04-22T22:19:04.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:04 smithi047 ceph-mon[21846]: Reconfiguring daemon alertmanager.smithi047 on smithi047 2024-04-22T22:19:04.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:04 smithi113 ceph-mon[24083]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:04.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:04 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:04.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:04 smithi113 ceph-mon[24083]: Reconfiguring alertmanager.smithi047 (dependencies changed)... 2024-04-22T22:19:04.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:04 smithi113 ceph-mon[24083]: Reconfiguring daemon alertmanager.smithi047 on smithi047 2024-04-22T22:19:05.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:05 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T22:19:05.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:05 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:05.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:05 smithi047 ceph-mon[21846]: Deploying daemon osd.0 on smithi047 2024-04-22T22:19:05.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:05 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T22:19:05.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:05 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:05.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:05 smithi113 ceph-mon[24083]: Deploying daemon osd.0 on smithi047 2024-04-22T22:19:06.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:06 smithi047 ceph-mon[21846]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:06.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:06 smithi113 ceph-mon[24083]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:08.382 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:08 smithi047 ceph-mon[21846]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:08.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:08 smithi113 ceph-mon[24083]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:10.558 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:10 smithi047 ceph-mon[21846]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:10.558 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:10.558 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:10 smithi047 ceph-mon[21846]: Reconfiguring crash.smithi047 (monmap changed)... 2024-04-22T22:19:10.558 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi047", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T22:19:10.558 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:10.558 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:10 smithi047 ceph-mon[21846]: Reconfiguring daemon crash.smithi047 on smithi047 2024-04-22T22:19:10.736 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:10 smithi113 ceph-mon[24083]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:10.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:10.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:10 smithi113 ceph-mon[24083]: Reconfiguring crash.smithi047 (monmap changed)... 2024-04-22T22:19:10.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi047", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T22:19:10.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:10.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:10 smithi113 ceph-mon[24083]: Reconfiguring daemon crash.smithi047 on smithi047 2024-04-22T22:19:12.615 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:12 smithi047 ceph-mon[21846]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:12.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:12 smithi113 ceph-mon[24083]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:14.273 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:14 smithi047 ceph-mon[21846]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:14.486 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:14 smithi113 ceph-mon[24083]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:15.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:15.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:19:15.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:19:15.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:15.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:19:15.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:19:16.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:16 smithi113 ceph-mon[24083]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:16.752 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:16 smithi047 ceph-mon[21846]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:18.117 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:17 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:18.117 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:17 smithi047 ceph-mon[21846]: Reconfiguring grafana.smithi047 (dependencies changed)... 2024-04-22T22:19:18.117 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:17 smithi047 ceph-mon[21846]: Reconfiguring daemon grafana.smithi047 on smithi047 2024-04-22T22:19:18.117 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:17 smithi047 ceph-mon[21846]: from='osd.0 [v2:172.21.15.47:6802/2475417589,v1:172.21.15.47:6803/2475417589]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T22:19:18.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:17 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:18.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:17 smithi113 ceph-mon[24083]: Reconfiguring grafana.smithi047 (dependencies changed)... 2024-04-22T22:19:18.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:17 smithi113 ceph-mon[24083]: Reconfiguring daemon grafana.smithi047 on smithi047 2024-04-22T22:19:18.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:17 smithi113 ceph-mon[24083]: from='osd.0 [v2:172.21.15.47:6802/2475417589,v1:172.21.15.47:6803/2475417589]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T22:19:19.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:18 smithi047 ceph-mon[21846]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:19.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:18 smithi047 ceph-mon[21846]: from='osd.0 [v2:172.21.15.47:6802/2475417589,v1:172.21.15.47:6803/2475417589]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T22:19:19.134 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:18 smithi047 ceph-mon[21846]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T22:19:19.134 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:18 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:19:19.134 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:18 smithi047 ceph-mon[21846]: from='osd.0 [v2:172.21.15.47:6802/2475417589,v1:172.21.15.47:6803/2475417589]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-04-22T22:19:19.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:18 smithi113 ceph-mon[24083]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:19.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:18 smithi113 ceph-mon[24083]: from='osd.0 [v2:172.21.15.47:6802/2475417589,v1:172.21.15.47:6803/2475417589]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T22:19:19.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:18 smithi113 ceph-mon[24083]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T22:19:19.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:18 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:19:19.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:18 smithi113 ceph-mon[24083]: from='osd.0 [v2:172.21.15.47:6802/2475417589,v1:172.21.15.47:6803/2475417589]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-04-22T22:19:19.557 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 0 on host 'smithi047' 2024-04-22T22:19:20.113 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:19 smithi047 ceph-mon[21846]: from='osd.0 [v2:172.21.15.47:6802/2475417589,v1:172.21.15.47:6803/2475417589]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-04-22T22:19:20.114 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:19 smithi047 ceph-mon[21846]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T22:19:20.114 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:19:20.114 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:19:20.114 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:20.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:19 smithi113 ceph-mon[24083]: from='osd.0 [v2:172.21.15.47:6802/2475417589,v1:172.21.15.47:6803/2475417589]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-04-22T22:19:20.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:19 smithi113 ceph-mon[24083]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T22:19:20.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:19 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:19:20.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:19 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:19:20.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:19 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:20.426 DEBUG:teuthology.orchestra.run.smithi047:osd.0> sudo journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T22:19:20.429 INFO:tasks.cephadm:Deploying osd.1 on smithi047 with /dev/vg_nvme/lv_3... 2024-04-22T22:19:20.429 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T22:19:21.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:20 smithi047 ceph-mon[21846]: purged_snaps scrub starts 2024-04-22T22:19:21.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:20 smithi047 ceph-mon[21846]: purged_snaps scrub ok 2024-04-22T22:19:21.009 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:20 smithi047 ceph-mon[21846]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:21.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:19:21.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:20 smithi047 ceph-mon[21846]: osd.0 [v2:172.21.15.47:6802/2475417589,v1:172.21.15.47:6803/2475417589] boot 2024-04-22T22:19:21.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:20 smithi047 ceph-mon[21846]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T22:19:21.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:19:21.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:20 smithi113 ceph-mon[24083]: purged_snaps scrub starts 2024-04-22T22:19:21.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:20 smithi113 ceph-mon[24083]: purged_snaps scrub ok 2024-04-22T22:19:21.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:20 smithi113 ceph-mon[24083]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:19:21.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:20 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:19:21.238 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:20 smithi113 ceph-mon[24083]: osd.0 [v2:172.21.15.47:6802/2475417589,v1:172.21.15.47:6803/2475417589] boot 2024-04-22T22:19:21.238 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:20 smithi113 ceph-mon[24083]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T22:19:21.238 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:20 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:19:21.884 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:21 smithi047 ceph-mon[21846]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T22:19:22.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:21 smithi113 ceph-mon[24083]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T22:19:23.086 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:22 smithi047 ceph-mon[21846]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:23.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:22 smithi113 ceph-mon[24083]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:24.188 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T22:19:24.188 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T22:19:24.188 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2024-04-22T22:19:24.188 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2024-04-22T22:19:24.189 INFO:teuthology.orchestra.run.smithi047.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00839467 s, 1.2 GB/s 2024-04-22T22:19:24.189 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 2024-04-22T22:19:24.189 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2024-04-22T22:19:24.383 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:24 smithi047 ceph-mon[21846]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:24.487 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:24 smithi113 ceph-mon[24083]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:25.063 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi047:vg_nvme/lv_3 2024-04-22T22:19:26.633 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:26 smithi047 ceph-mon[21846]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:26.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:26 smithi113 ceph-mon[24083]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:27.776 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:27 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:27.777 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:27 smithi113 ceph-mon[24083]: Reconfiguring crash.smithi113 (monmap changed)... 2024-04-22T22:19:27.777 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:27 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T22:19:27.777 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:27 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:27.777 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:27 smithi113 ceph-mon[24083]: Reconfiguring daemon crash.smithi113 on smithi113 2024-04-22T22:19:27.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:27 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:27.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:27 smithi047 ceph-mon[21846]: Reconfiguring crash.smithi113 (monmap changed)... 2024-04-22T22:19:27.883 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:27 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T22:19:27.884 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:27 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:27.884 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:27 smithi047 ceph-mon[21846]: Reconfiguring daemon crash.smithi113 on smithi113 2024-04-22T22:19:28.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:28 smithi047 ceph-mon[21846]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:28.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:28 smithi047 ceph-mon[21846]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:19:28.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:19:28.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:19:28.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:28.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:28.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:28 smithi047 ceph-mon[21846]: Reconfiguring mgr.smithi113.qvxssq (monmap changed)... 2024-04-22T22:19:28.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.qvxssq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:19:28.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:19:28.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:28.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:28 smithi047 ceph-mon[21846]: Reconfiguring daemon mgr.smithi113.qvxssq on smithi113 2024-04-22T22:19:28.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:28 smithi113 ceph-mon[24083]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:28.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:28 smithi113 ceph-mon[24083]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:19:28.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:28 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:19:28.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:28 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:19:28.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:28 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:28.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:28 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:28.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:28 smithi113 ceph-mon[24083]: Reconfiguring mgr.smithi113.qvxssq (monmap changed)... 2024-04-22T22:19:28.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:28 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.qvxssq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:19:28.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:28 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:19:28.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:28 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:28.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:28 smithi113 ceph-mon[24083]: Reconfiguring daemon mgr.smithi113.qvxssq on smithi113 2024-04-22T22:19:30.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:30 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:30.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:30 smithi047 ceph-mon[21846]: Reconfiguring mon.smithi113 (monmap changed)... 2024-04-22T22:19:30.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:30 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:19:30.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:30 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:19:30.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:30 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:30.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:30 smithi047 ceph-mon[21846]: Reconfiguring daemon mon.smithi113 on smithi113 2024-04-22T22:19:30.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:30 smithi047 ceph-mon[21846]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:30.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:30 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2927222957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96a3a82d-1779-4fe6-96da-9071bbd29214"}]: dispatch 2024-04-22T22:19:30.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:30 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2927222957' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96a3a82d-1779-4fe6-96da-9071bbd29214"}]': finished 2024-04-22T22:19:30.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:30 smithi047 ceph-mon[21846]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T22:19:30.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:30 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:19:30.439 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:30 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:30.439 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:30 smithi113 ceph-mon[24083]: Reconfiguring mon.smithi113 (monmap changed)... 2024-04-22T22:19:30.440 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:30 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:19:30.440 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:30 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:19:30.440 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:30 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:30.440 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:30 smithi113 ceph-mon[24083]: Reconfiguring daemon mon.smithi113 on smithi113 2024-04-22T22:19:30.440 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:30 smithi113 ceph-mon[24083]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:30.440 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:30 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2927222957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96a3a82d-1779-4fe6-96da-9071bbd29214"}]: dispatch 2024-04-22T22:19:30.440 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:30 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2927222957' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96a3a82d-1779-4fe6-96da-9071bbd29214"}]': finished 2024-04-22T22:19:30.440 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:30 smithi113 ceph-mon[24083]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T22:19:30.440 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:30 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:19:31.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:31 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:31.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:31 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T22:19:31.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:31 smithi047 ceph-mon[21846]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T22:19:31.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:31 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2428222264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:19:31.906 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:31 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:31.906 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:31 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T22:19:31.906 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:31 smithi113 ceph-mon[24083]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T22:19:31.906 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:31 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2428222264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:19:32.825 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:32 smithi047 ceph-mon[21846]: pgmap v37: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:32.987 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:32 smithi113 ceph-mon[24083]: pgmap v37: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:34.127 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:34 smithi047 ceph-mon[21846]: pgmap v38: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:34.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:34 smithi113 ceph-mon[24083]: pgmap v38: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:36.580 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:36 smithi047 ceph-mon[21846]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:36.580 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:36 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T22:19:36.580 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:36 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:36.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:36 smithi113 ceph-mon[24083]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:36.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:36 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T22:19:36.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:36 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:37.583 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:37 smithi047 ceph-mon[21846]: Deploying daemon osd.1 on smithi047 2024-04-22T22:19:37.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:37 smithi113 ceph-mon[24083]: Deploying daemon osd.1 on smithi047 2024-04-22T22:19:38.384 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:38 smithi047 ceph-mon[21846]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:38.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:38 smithi113 ceph-mon[24083]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:40.449 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:40 smithi047 ceph-mon[21846]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:40.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:40 smithi113 ceph-mon[24083]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:42.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:42 smithi113 ceph-mon[24083]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:42.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:42 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:42.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:42 smithi047 ceph-mon[21846]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:42.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:42 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:44.166 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:44 smithi047 ceph-mon[21846]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:44.200 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 1 on host 'smithi047' 2024-04-22T22:19:44.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:44 smithi113 ceph-mon[24083]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:44.828 DEBUG:teuthology.orchestra.run.smithi047:osd.1> sudo journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T22:19:44.830 INFO:tasks.cephadm:Deploying osd.2 on smithi047 with /dev/vg_nvme/lv_2... 2024-04-22T22:19:44.830 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T22:19:45.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:45.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.47:9093"}]: dispatch 2024-04-22T22:19:45.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.47:9093"}]: dispatch 2024-04-22T22:19:45.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:45.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.47:3000"}]: dispatch 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.47:3000"}]: dispatch 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.47:9095"}]: dispatch 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.47:9095"}]: dispatch 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:19:45.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:45 smithi113 ceph-mon[24083]: from='osd.1 [v2:172.21.15.47:6810/3406095492,v1:172.21.15.47:6811/3406095492]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T22:19:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.47:9093"}]: dispatch 2024-04-22T22:19:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.47:9093"}]: dispatch 2024-04-22T22:19:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T22:19:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T22:19:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.47:3000"}]: dispatch 2024-04-22T22:19:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.47:3000"}]: dispatch 2024-04-22T22:19:45.507 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:45.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:19:45.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:19:45.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.47:9095"}]: dispatch 2024-04-22T22:19:45.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.47:9095"}]: dispatch 2024-04-22T22:19:45.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:45.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:19:45.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:45.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:19:45.508 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:45 smithi047 ceph-mon[21846]: from='osd.1 [v2:172.21.15.47:6810/3406095492,v1:172.21.15.47:6811/3406095492]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T22:19:46.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:46 smithi113 ceph-mon[24083]: from='osd.1 [v2:172.21.15.47:6810/3406095492,v1:172.21.15.47:6811/3406095492]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T22:19:46.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:46 smithi113 ceph-mon[24083]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T22:19:46.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:46 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:19:46.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:46 smithi113 ceph-mon[24083]: from='osd.1 [v2:172.21.15.47:6810/3406095492,v1:172.21.15.47:6811/3406095492]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-04-22T22:19:46.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:46 smithi113 ceph-mon[24083]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:46.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:46 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:46.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:46 smithi047 ceph-mon[21846]: from='osd.1 [v2:172.21.15.47:6810/3406095492,v1:172.21.15.47:6811/3406095492]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T22:19:46.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:46 smithi047 ceph-mon[21846]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T22:19:46.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:46 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:19:46.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:46 smithi047 ceph-mon[21846]: from='osd.1 [v2:172.21.15.47:6810/3406095492,v1:172.21.15.47:6811/3406095492]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-04-22T22:19:46.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:46 smithi047 ceph-mon[21846]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:46.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:46 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:46.635 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:19:46 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1[35133]: 2024-04-22T22:19:46.259+0000 7f28368c0700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T22:19:46.635 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:19:46 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1[35133]: 2024-04-22T22:19:46.262+0000 7f283325b700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:19:47.385 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:47 smithi047 ceph-mon[21846]: from='osd.1 [v2:172.21.15.47:6810/3406095492,v1:172.21.15.47:6811/3406095492]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-04-22T22:19:47.386 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:47 smithi047 ceph-mon[21846]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T22:19:47.386 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:47 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:19:47.386 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:47 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:19:47.521 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T22:19:47.521 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T22:19:47.522 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2024-04-22T22:19:47.522 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2024-04-22T22:19:47.522 INFO:teuthology.orchestra.run.smithi047.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00954179 s, 1.1 GB/s 2024-04-22T22:19:47.522 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2024-04-22T22:19:47.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:47 smithi113 ceph-mon[24083]: from='osd.1 [v2:172.21.15.47:6810/3406095492,v1:172.21.15.47:6811/3406095492]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-04-22T22:19:47.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:47 smithi113 ceph-mon[24083]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T22:19:47.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:47 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:19:47.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:47 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:19:48.176 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi047:vg_nvme/lv_2 2024-04-22T22:19:48.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:48 smithi047 ceph-mon[21846]: purged_snaps scrub starts 2024-04-22T22:19:48.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:48 smithi047 ceph-mon[21846]: purged_snaps scrub ok 2024-04-22T22:19:48.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:48 smithi047 ceph-mon[21846]: osd.1 [v2:172.21.15.47:6810/3406095492,v1:172.21.15.47:6811/3406095492] boot 2024-04-22T22:19:48.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:48 smithi047 ceph-mon[21846]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T22:19:48.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:48 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:19:48.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:48 smithi047 ceph-mon[21846]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:48.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:48 smithi113 ceph-mon[24083]: purged_snaps scrub starts 2024-04-22T22:19:48.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:48 smithi113 ceph-mon[24083]: purged_snaps scrub ok 2024-04-22T22:19:48.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:48 smithi113 ceph-mon[24083]: osd.1 [v2:172.21.15.47:6810/3406095492,v1:172.21.15.47:6811/3406095492] boot 2024-04-22T22:19:48.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:48 smithi113 ceph-mon[24083]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T22:19:48.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:48 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:19:48.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:48 smithi113 ceph-mon[24083]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:19:49.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:49 smithi047 ceph-mon[21846]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T22:19:49.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:49 smithi113 ceph-mon[24083]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T22:19:50.528 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:50 smithi047 ceph-mon[21846]: pgmap v50: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:19:50.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:50 smithi113 ceph-mon[24083]: pgmap v50: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:19:51.634 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:51 smithi047 ceph-mon[21846]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:19:51.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:51 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:19:51.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:51 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:19:51.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:51 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:51.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:51 smithi113 ceph-mon[24083]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:19:51.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:51 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:19:51.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:51 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:19:51.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:51 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:19:52.461 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:52 smithi047 ceph-mon[21846]: pgmap v51: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:19:52.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:52 smithi113 ceph-mon[24083]: pgmap v51: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:19:54.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:54 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2414425586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0de44f6b-8fea-441f-9de7-dc6d1156164e"}]: dispatch 2024-04-22T22:19:54.377 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:54 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2414425586' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0de44f6b-8fea-441f-9de7-dc6d1156164e"}]': finished 2024-04-22T22:19:54.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:54 smithi047 ceph-mon[21846]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T22:19:54.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:54 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:19:54.378 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:54 smithi047 ceph-mon[21846]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:19:54.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:54 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2414425586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0de44f6b-8fea-441f-9de7-dc6d1156164e"}]: dispatch 2024-04-22T22:19:54.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:54 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2414425586' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0de44f6b-8fea-441f-9de7-dc6d1156164e"}]': finished 2024-04-22T22:19:54.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:54 smithi113 ceph-mon[24083]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T22:19:54.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:54 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:19:54.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:54 smithi113 ceph-mon[24083]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:19:55.385 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:55 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:55.385 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:55 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/3250914272' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:19:55.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:55 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:19:55.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:55 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/3250914272' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:19:56.337 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:56 smithi047 ceph-mon[21846]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:19:56.488 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:56 smithi113 ceph-mon[24083]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:19:58.385 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:19:58 smithi047 ceph-mon[21846]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:19:58.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:19:58 smithi113 ceph-mon[24083]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:00.253 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: Detected new or changed devices on smithi047 2024-04-22T22:20:00.253 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:00.253 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:00.253 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:00.253 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: Adjusting osd_memory_target on smithi047 to 6434M 2024-04-22T22:20:00.253 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:00.253 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:00.253 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:20:00.253 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:00.254 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:20:00.254 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:00.254 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:00 smithi113 ceph-mon[24083]: overall HEALTH_OK 2024-04-22T22:20:00.366 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: Detected new or changed devices on smithi047 2024-04-22T22:20:00.367 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:00.367 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:00.367 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:00.367 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: Adjusting osd_memory_target on smithi047 to 6434M 2024-04-22T22:20:00.367 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:00.367 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:00.367 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:20:00.367 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:00.367 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:20:00.367 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:00.367 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:00 smithi047 ceph-mon[21846]: overall HEALTH_OK 2024-04-22T22:20:01.955 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:01 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:01.955 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:01 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T22:20:01.955 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:01 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:02.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:01 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:02.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:01 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T22:20:02.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:01 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:03.135 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:02 smithi047 ceph-mon[21846]: Deploying daemon osd.2 on smithi047 2024-04-22T22:20:03.135 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:02 smithi047 ceph-mon[21846]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:03.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:02 smithi113 ceph-mon[24083]: Deploying daemon osd.2 on smithi047 2024-04-22T22:20:03.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:02 smithi113 ceph-mon[24083]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:04.385 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:04 smithi047 ceph-mon[21846]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:04.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:04 smithi113 ceph-mon[24083]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:06.635 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:06 smithi047 ceph-mon[21846]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:06.739 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:06 smithi113 ceph-mon[24083]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:08.102 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:08 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:08.102 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:08 smithi047 ceph-mon[21846]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:08.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:08 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:08.489 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:08 smithi113 ceph-mon[24083]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:10.885 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:10 smithi047 ceph-mon[21846]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:10.885 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:10.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:10 smithi113 ceph-mon[24083]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:10.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:12.837 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:12 smithi113 ceph-mon[24083]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:12.837 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:12.837 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:12.837 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:20:12.837 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:12.838 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:20:12.847 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:12 smithi047 ceph-mon[21846]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:12.847 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:12.847 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:12.847 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:20:12.847 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:12.847 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:20:13.767 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:13 smithi047 ceph-mon[21846]: from='osd.2 [v2:172.21.15.47:6818/2629230070,v1:172.21.15.47:6819/2629230070]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T22:20:13.767 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:13 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:13.802 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 2 on host 'smithi047' 2024-04-22T22:20:13.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:13 smithi113 ceph-mon[24083]: from='osd.2 [v2:172.21.15.47:6818/2629230070,v1:172.21.15.47:6819/2629230070]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T22:20:13.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:13 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:14.543 DEBUG:teuthology.orchestra.run.smithi047:osd.2> sudo journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T22:20:14.545 INFO:tasks.cephadm:Deploying osd.3 on smithi047 with /dev/vg_nvme/lv_1... 2024-04-22T22:20:14.546 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T22:20:14.885 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:14 smithi047 ceph-mon[21846]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:14.885 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:14 smithi047 ceph-mon[21846]: from='osd.2 [v2:172.21.15.47:6818/2629230070,v1:172.21.15.47:6819/2629230070]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T22:20:14.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:14 smithi047 ceph-mon[21846]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T22:20:14.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:14 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:20:14.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:14 smithi047 ceph-mon[21846]: from='osd.2 [v2:172.21.15.47:6818/2629230070,v1:172.21.15.47:6819/2629230070]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-04-22T22:20:14.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:14 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:14.886 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:20:14 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2[39009]: 2024-04-22T22:20:14.600+0000 7f83447ce700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T22:20:14.886 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:20:14 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2[39009]: 2024-04-22T22:20:14.604+0000 7f833d161700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:20:14.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:14 smithi113 ceph-mon[24083]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:14.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:14 smithi113 ceph-mon[24083]: from='osd.2 [v2:172.21.15.47:6818/2629230070,v1:172.21.15.47:6819/2629230070]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T22:20:14.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:14 smithi113 ceph-mon[24083]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T22:20:14.990 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:14 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:20:14.990 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:14 smithi113 ceph-mon[24083]: from='osd.2 [v2:172.21.15.47:6818/2629230070,v1:172.21.15.47:6819/2629230070]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-04-22T22:20:14.990 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:14 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:15.843 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:15 smithi047 ceph-mon[21846]: from='osd.2 [v2:172.21.15.47:6818/2629230070,v1:172.21.15.47:6819/2629230070]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-04-22T22:20:15.843 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:15 smithi047 ceph-mon[21846]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T22:20:15.843 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:20:15.844 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:20:15.844 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:20:15.844 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:20:15.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:15 smithi113 ceph-mon[24083]: from='osd.2 [v2:172.21.15.47:6818/2629230070,v1:172.21.15.47:6819/2629230070]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-04-22T22:20:15.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:15 smithi113 ceph-mon[24083]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T22:20:15.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:20:15.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:20:15.990 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:20:15.990 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:20:16.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:16 smithi047 ceph-mon[21846]: purged_snaps scrub starts 2024-04-22T22:20:16.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:16 smithi047 ceph-mon[21846]: purged_snaps scrub ok 2024-04-22T22:20:16.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:16 smithi047 ceph-mon[21846]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:16.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:16 smithi047 ceph-mon[21846]: osd.2 [v2:172.21.15.47:6818/2629230070,v1:172.21.15.47:6819/2629230070] boot 2024-04-22T22:20:16.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:16 smithi047 ceph-mon[21846]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T22:20:16.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:20:16.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:16 smithi113 ceph-mon[24083]: purged_snaps scrub starts 2024-04-22T22:20:16.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:16 smithi113 ceph-mon[24083]: purged_snaps scrub ok 2024-04-22T22:20:16.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:16 smithi113 ceph-mon[24083]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:20:16.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:16 smithi113 ceph-mon[24083]: osd.2 [v2:172.21.15.47:6818/2629230070,v1:172.21.15.47:6819/2629230070] boot 2024-04-22T22:20:16.989 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:16 smithi113 ceph-mon[24083]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T22:20:16.990 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:16 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:20:17.086 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T22:20:17.086 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T22:20:17.086 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2024-04-22T22:20:17.087 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2024-04-22T22:20:17.087 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0100764 s, 1.0 GB/s 2024-04-22T22:20:17.087 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2024-04-22T22:20:17.722 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi047:vg_nvme/lv_1 2024-04-22T22:20:18.135 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:17 smithi047 ceph-mon[21846]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T22:20:18.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:17 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T22:20:18.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:17 smithi113 ceph-mon[24083]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T22:20:18.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:17 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T22:20:18.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:18 smithi047 ceph-mon[21846]: pgmap v69: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:18.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:18 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T22:20:18.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:18 smithi047 ceph-mon[21846]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T22:20:18.886 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:18 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T22:20:19.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:18 smithi113 ceph-mon[24083]: pgmap v69: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:19.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:18 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T22:20:19.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:18 smithi113 ceph-mon[24083]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T22:20:19.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:18 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T22:20:19.968 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T22:20:19.968 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:19 smithi047 ceph-mon[21846]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T22:20:20.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:19 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T22:20:20.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:19 smithi113 ceph-mon[24083]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T22:20:20.250 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:20:20 smithi047 sudo[40450]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T22:20:20.251 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:20:20 smithi047 sudo[40450]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:20:20.251 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:20:20 smithi047 sudo[40450]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:20:20.829 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:20:20 smithi047 sudo[40485]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T22:20:20.829 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:20:20 smithi047 sudo[40485]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:20:20.829 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:20:20 smithi047 sudo[40485]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:20:21.108 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:20:21 smithi047 sudo[40524]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T22:20:21.109 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:20:21 smithi047 sudo[40524]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:20:21.109 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:20:21 smithi047 sudo[40524]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:20:21.109 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:20 smithi047 ceph-mon[21846]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:21.109 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:20 smithi047 ceph-mon[21846]: osdmap e23: 3 total, 3 up, 3 in 2024-04-22T22:20:21.109 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:21.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:20 smithi113 ceph-mon[24083]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:21.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:20 smithi113 ceph-mon[24083]: osdmap e23: 3 total, 3 up, 3 in 2024-04-22T22:20:21.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:20 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:21.635 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:20:21 smithi047 sudo[40675]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T22:20:21.636 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:20:21 smithi047 sudo[40675]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:20:21.636 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:20:21 smithi047 sudo[40675]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:20:22.259 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:20:21 smithi047 sudo[40730]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T22:20:22.259 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:20:21 smithi047 sudo[40730]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:20:22.259 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:20:22 smithi047 sudo[40730]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:20:22.616 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:20:22 smithi047 sudo[40751]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T22:20:22.616 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:20:22 smithi047 sudo[40751]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:20:22.616 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:20:22 smithi047 sudo[40751]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:20:22.617 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:22 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:22.617 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:22 smithi047 ceph-mon[21846]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:22.617 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:22 smithi047 ceph-mon[21846]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:20:22.617 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:22 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:20:22.617 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:22 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:20:22.617 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:22 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:22.617 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:22 smithi047 ceph-mon[21846]: mgrmap e19: smithi047.uutpnj(active, since 2m), standbys: smithi113.qvxssq 2024-04-22T22:20:22.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:22 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:22.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:22 smithi113 ceph-mon[24083]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:22.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:22 smithi113 ceph-mon[24083]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:20:22.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:22 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:20:22.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:22 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:20:22.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:22 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:22.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:22 smithi113 ceph-mon[24083]: mgrmap e19: smithi047.uutpnj(active, since 2m), standbys: smithi113.qvxssq 2024-04-22T22:20:23.204 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:22 smithi047 sudo[40774]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T22:20:23.205 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:22 smithi047 sudo[40774]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:20:23.205 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:22 smithi047 sudo[40774]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:20:23.531 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:23 smithi047 sudo[40787]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T22:20:23.531 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:23 smithi047 sudo[40787]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:20:23.531 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:23 smithi047 sudo[40787]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:20:23.856 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 sudo[26392]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T22:20:23.857 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 sudo[26392]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:20:23.857 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 sudo[26392]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:20:24.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:23 smithi047 ceph-mon[21846]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:20:24.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:23 smithi047 ceph-mon[21846]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:20:24.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:23 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-04-22T22:20:24.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:23 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:20:24.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:23 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-04-22T22:20:24.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:23 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:20:24.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 sudo[26395]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T22:20:24.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 sudo[26395]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:20:24.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 sudo[26395]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:20:24.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 ceph-mon[24083]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:20:24.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 ceph-mon[24083]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:20:24.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-04-22T22:20:24.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:20:24.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-04-22T22:20:24.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:23 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-22T22:20:25.122 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:24 smithi047 ceph-mon[21846]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:25.123 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:24 smithi047 ceph-mon[21846]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:20:25.123 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:24 smithi047 ceph-mon[21846]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:20:25.123 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:24 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:25.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:24 smithi113 ceph-mon[24083]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:25.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:24 smithi113 ceph-mon[24083]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:20:25.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:24 smithi113 ceph-mon[24083]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:20:25.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:24 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:26.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:26.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:26.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:26.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:26.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:25 smithi047 ceph-mon[21846]: Adjusting osd_memory_target on smithi047 to 4289M 2024-04-22T22:20:26.137 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:26.137 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:26.137 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:20:26.137 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:26.137 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:25 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:20:26.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:25 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:26.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:25 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:26.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:25 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:26.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:25 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:26.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:25 smithi113 ceph-mon[24083]: Adjusting osd_memory_target on smithi047 to 4289M 2024-04-22T22:20:26.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:25 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:26.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:25 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:26.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:25 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:20:26.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:25 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:26.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:25 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:20:27.239 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:27 smithi113 ceph-mon[24083]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:27.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:27 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/4218376264' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0eff041-a458-45ed-ae2e-288d257c7d78"}]: dispatch 2024-04-22T22:20:27.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:27 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/4218376264' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0eff041-a458-45ed-ae2e-288d257c7d78"}]': finished 2024-04-22T22:20:27.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:27 smithi113 ceph-mon[24083]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T22:20:27.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:27 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:27.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:27 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/3155241045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:20:27.305 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:27 smithi047 ceph-mon[21846]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:27.305 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:27 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/4218376264' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0eff041-a458-45ed-ae2e-288d257c7d78"}]: dispatch 2024-04-22T22:20:27.305 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:27 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/4218376264' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0eff041-a458-45ed-ae2e-288d257c7d78"}]': finished 2024-04-22T22:20:27.305 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:27 smithi047 ceph-mon[21846]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T22:20:27.305 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:27 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:27.305 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:27 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/3155241045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:20:29.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:29 smithi047 ceph-mon[21846]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:29.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:29 smithi113 ceph-mon[24083]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:31.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:31 smithi113 ceph-mon[24083]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-22T22:20:31.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:31 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:31.240 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:31 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:31.289 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:31 smithi047 ceph-mon[21846]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-22T22:20:31.290 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:31 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:31.290 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:31 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:33.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:33 smithi047 ceph-mon[21846]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-22T22:20:33.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:33 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T22:20:33.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:33 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:33.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:33 smithi113 ceph-mon[24083]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-22T22:20:33.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:33 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T22:20:33.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:33 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:34.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:34 smithi047 ceph-mon[21846]: Deploying daemon osd.3 on smithi047 2024-04-22T22:20:34.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:34 smithi113 ceph-mon[24083]: Deploying daemon osd.3 on smithi047 2024-04-22T22:20:35.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:35 smithi047 ceph-mon[21846]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-22T22:20:35.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:35 smithi113 ceph-mon[24083]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-22T22:20:37.275 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:37 smithi047 ceph-mon[21846]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-22T22:20:37.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:37 smithi113 ceph-mon[24083]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-22T22:20:39.137 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:39 smithi047 ceph-mon[21846]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 4 op/s 2024-04-22T22:20:39.137 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:39 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:39.137 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:39 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:20:39.137 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:39 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:39.137 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:39 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:20:39.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:39 smithi113 ceph-mon[24083]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 4 op/s 2024-04-22T22:20:39.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:39 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:39.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:39 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:20:39.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:39 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:39.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:39 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:20:40.136 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:40 smithi047 ceph-mon[21846]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T22:20:40.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:40 smithi113 ceph-mon[24083]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T22:20:41.345 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 3 on host 'smithi047' 2024-04-22T22:20:41.636 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:41 smithi047 ceph-mon[21846]: from='osd.3 [v2:172.21.15.47:6826/1765039522,v1:172.21.15.47:6827/1765039522]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T22:20:41.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:41 smithi113 ceph-mon[24083]: from='osd.3 [v2:172.21.15.47:6826/1765039522,v1:172.21.15.47:6827/1765039522]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T22:20:41.979 DEBUG:teuthology.orchestra.run.smithi047:osd.3> sudo journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T22:20:41.981 INFO:tasks.cephadm:Deploying osd.4 on smithi113 with /dev/vg_nvme/lv_4... 2024-04-22T22:20:41.981 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T22:20:42.556 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:42 smithi113 ceph-mon[24083]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:42.556 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:42 smithi113 ceph-mon[24083]: from='osd.3 [v2:172.21.15.47:6826/1765039522,v1:172.21.15.47:6827/1765039522]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T22:20:42.556 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:42 smithi113 ceph-mon[24083]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T22:20:42.556 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:42 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:42.556 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:42 smithi113 ceph-mon[24083]: from='osd.3 [v2:172.21.15.47:6826/1765039522,v1:172.21.15.47:6827/1765039522]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-04-22T22:20:42.556 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:42 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:42.557 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:42 smithi113 ceph-mon[24083]: from='osd.3 [v2:172.21.15.47:6826/1765039522,v1:172.21.15.47:6827/1765039522]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-04-22T22:20:42.557 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:42 smithi113 ceph-mon[24083]: osdmap e26: 4 total, 3 up, 4 in 2024-04-22T22:20:42.557 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:42 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:42.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:42 smithi047 ceph-mon[21846]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:42.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:42 smithi047 ceph-mon[21846]: from='osd.3 [v2:172.21.15.47:6826/1765039522,v1:172.21.15.47:6827/1765039522]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T22:20:42.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:42 smithi047 ceph-mon[21846]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T22:20:42.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:42 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:42.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:42 smithi047 ceph-mon[21846]: from='osd.3 [v2:172.21.15.47:6826/1765039522,v1:172.21.15.47:6827/1765039522]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-04-22T22:20:42.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:42 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:42.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:42 smithi047 ceph-mon[21846]: from='osd.3 [v2:172.21.15.47:6826/1765039522,v1:172.21.15.47:6827/1765039522]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-04-22T22:20:42.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:42 smithi047 ceph-mon[21846]: osdmap e26: 4 total, 3 up, 4 in 2024-04-22T22:20:42.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:42 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:42.638 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:20:42 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3[42883]: 2024-04-22T22:20:42.336+0000 7f51e7bf0700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T22:20:42.638 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:20:42 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3[42883]: 2024-04-22T22:20:42.340+0000 7f51e458b700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:20:43.350 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:43 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:43.582 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:43 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:44.088 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T22:20:44.088 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T22:20:44.089 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-04-22T22:20:44.089 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-04-22T22:20:44.089 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00820203 s, 1.3 GB/s 2024-04-22T22:20:44.089 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-04-22T22:20:44.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:44 smithi047 ceph-mon[21846]: purged_snaps scrub starts 2024-04-22T22:20:44.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:44 smithi047 ceph-mon[21846]: purged_snaps scrub ok 2024-04-22T22:20:44.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:44 smithi047 ceph-mon[21846]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:44.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:44 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:44.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:44 smithi047 ceph-mon[21846]: osd.3 [v2:172.21.15.47:6826/1765039522,v1:172.21.15.47:6827/1765039522] boot 2024-04-22T22:20:44.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:44 smithi047 ceph-mon[21846]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T22:20:44.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:44 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:44.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:44 smithi113 ceph-mon[24083]: purged_snaps scrub starts 2024-04-22T22:20:44.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:44 smithi113 ceph-mon[24083]: purged_snaps scrub ok 2024-04-22T22:20:44.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:44 smithi113 ceph-mon[24083]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:20:44.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:44 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:44.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:44 smithi113 ceph-mon[24083]: osd.3 [v2:172.21.15.47:6826/1765039522,v1:172.21.15.47:6827/1765039522] boot 2024-04-22T22:20:44.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:44 smithi113 ceph-mon[24083]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T22:20:44.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:44 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:20:44.725 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi113:vg_nvme/lv_4 2024-04-22T22:20:46.261 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:46 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:46.261 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:46 smithi113 ceph-mon[24083]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:46.261 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:46 smithi113 ceph-mon[24083]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T22:20:46.483 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:46 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:46.483 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:46 smithi047 ceph-mon[21846]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:46.483 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:46 smithi047 ceph-mon[21846]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T22:20:47.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:47 smithi113 ceph-mon[24083]: from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:20:47.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:47 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:20:47.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:47 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:20:47.326 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:47 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:47.538 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:47 smithi047 ceph-mon[21846]: from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:20:47.538 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:47 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:20:47.538 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:47 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:20:47.538 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:47 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:48.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:48 smithi113 ceph-mon[24083]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:48.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:48 smithi047 ceph-mon[21846]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:49.484 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: Detected new or changed devices on smithi047 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: Adjusting osd_memory_target on smithi047 to 3217M 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/1235719082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f41cc9d-8c00-45b5-a9a6-b86071e2a081"}]: dispatch 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f41cc9d-8c00-45b5-a9a6-b86071e2a081"}]: dispatch 2024-04-22T22:20:49.485 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f41cc9d-8c00-45b5-a9a6-b86071e2a081"}]': finished 2024-04-22T22:20:49.486 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: osdmap e29: 5 total, 4 up, 5 in 2024-04-22T22:20:49.486 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:20:49.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: Detected new or changed devices on smithi047 2024-04-22T22:20:49.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:49.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:49.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:49.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:49.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:20:49.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: Adjusting osd_memory_target on smithi047 to 3217M 2024-04-22T22:20:49.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:49.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:49.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:20:49.492 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:49.492 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:20:49.492 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/1235719082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f41cc9d-8c00-45b5-a9a6-b86071e2a081"}]: dispatch 2024-04-22T22:20:49.492 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f41cc9d-8c00-45b5-a9a6-b86071e2a081"}]: dispatch 2024-04-22T22:20:49.492 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f41cc9d-8c00-45b5-a9a6-b86071e2a081"}]': finished 2024-04-22T22:20:49.492 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: osdmap e29: 5 total, 4 up, 5 in 2024-04-22T22:20:49.492 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:20:50.490 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:50 smithi113 ceph-mon[24083]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:50.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:50 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/987890929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:20:50.506 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:50 smithi047 ceph-mon[21846]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:50.506 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:50 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/987890929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:20:52.387 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:52 smithi047 ceph-mon[21846]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:52.648 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:52 smithi113 ceph-mon[24083]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:53.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:53 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:53.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:53 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:53.887 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:53 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:53.887 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:53 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:20:54.887 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:54 smithi047 ceph-mon[21846]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:54.914 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:54 smithi113 ceph-mon[24083]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:55.887 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:55 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T22:20:55.887 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:55 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:55.887 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:55 smithi047 ceph-mon[21846]: Deploying daemon osd.4 on smithi113 2024-04-22T22:20:55.991 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:55 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T22:20:55.991 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:55 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:20:55.991 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:55 smithi113 ceph-mon[24083]: Deploying daemon osd.4 on smithi113 2024-04-22T22:20:56.887 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:56 smithi047 ceph-mon[21846]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:56.897 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:56 smithi113 ceph-mon[24083]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:58.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:20:58 smithi113 ceph-mon[24083]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:20:58.887 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:20:58 smithi047 ceph-mon[21846]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:21:01.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:00 smithi113 ceph-mon[24083]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:21:01.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:01.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:01.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:01.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:01.290 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:00 smithi047 ceph-mon[21846]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:21:01.290 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:01.290 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:01.290 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:01.290 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:03.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:02 smithi113 ceph-mon[24083]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:21:03.299 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 4 on host 'smithi113' 2024-04-22T22:21:03.387 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:02 smithi047 ceph-mon[21846]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:21:03.945 DEBUG:teuthology.orchestra.run.smithi113:osd.4> sudo journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T22:21:03.948 INFO:tasks.cephadm:Deploying osd.5 on smithi113 with /dev/vg_nvme/lv_3... 2024-04-22T22:21:03.948 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T22:21:04.220 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:03 smithi113 ceph-mon[24083]: from='osd.4 [v2:172.21.15.113:6800/2948692086,v1:172.21.15.113:6801/2948692086]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T22:21:04.221 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:03 smithi113 ceph-mon[24083]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T22:21:04.221 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:03 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:04.387 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:03 smithi047 ceph-mon[21846]: from='osd.4 [v2:172.21.15.113:6800/2948692086,v1:172.21.15.113:6801/2948692086]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T22:21:04.387 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:03 smithi047 ceph-mon[21846]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T22:21:04.387 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:03 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:05.040 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:04 smithi113 ceph-mon[24083]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:21:05.040 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:04 smithi113 ceph-mon[24083]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T22:21:05.040 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:04 smithi113 ceph-mon[24083]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T22:21:05.040 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:04 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:21:05.040 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:04 smithi113 ceph-mon[24083]: from='osd.4 [v2:172.21.15.113:6800/2948692086,v1:172.21.15.113:6801/2948692086]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:05.041 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:04 smithi113 ceph-mon[24083]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:05.041 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:04 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:05.041 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:21:05 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:21:04.998+0000 7fa76697b700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T22:21:05.041 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:21:05 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:21:05.002+0000 7fa761b13700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:21:05.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:04 smithi047 ceph-mon[21846]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:21:05.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:04 smithi047 ceph-mon[21846]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T22:21:05.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:04 smithi047 ceph-mon[21846]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T22:21:05.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:04 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:21:05.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:04 smithi047 ceph-mon[21846]: from='osd.4 [v2:172.21.15.113:6800/2948692086,v1:172.21.15.113:6801/2948692086]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:05.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:04 smithi047 ceph-mon[21846]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:05.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:04 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:06.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:05 smithi113 ceph-mon[24083]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-22T22:21:06.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:05 smithi113 ceph-mon[24083]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T22:21:06.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:05 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:21:06.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:05 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:21:06.387 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:05 smithi047 ceph-mon[21846]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-22T22:21:06.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:05 smithi047 ceph-mon[21846]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T22:21:06.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:05 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:21:06.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:05 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:21:07.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:07 smithi113 ceph-mon[24083]: purged_snaps scrub starts 2024-04-22T22:21:07.242 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:07 smithi113 ceph-mon[24083]: purged_snaps scrub ok 2024-04-22T22:21:07.242 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:07 smithi113 ceph-mon[24083]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:21:07.242 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:07 smithi113 ceph-mon[24083]: osd.4 [v2:172.21.15.113:6800/2948692086,v1:172.21.15.113:6801/2948692086] boot 2024-04-22T22:21:07.242 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:07 smithi113 ceph-mon[24083]: osdmap e32: 5 total, 5 up, 5 in 2024-04-22T22:21:07.242 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:07 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:21:07.387 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:07 smithi047 ceph-mon[21846]: purged_snaps scrub starts 2024-04-22T22:21:07.387 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:07 smithi047 ceph-mon[21846]: purged_snaps scrub ok 2024-04-22T22:21:07.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:07 smithi047 ceph-mon[21846]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:21:07.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:07 smithi047 ceph-mon[21846]: osd.4 [v2:172.21.15.113:6800/2948692086,v1:172.21.15.113:6801/2948692086] boot 2024-04-22T22:21:07.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:07 smithi047 ceph-mon[21846]: osdmap e32: 5 total, 5 up, 5 in 2024-04-22T22:21:07.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:07 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:21:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T22:21:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T22:21:07.441 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-04-22T22:21:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-04-22T22:21:07.441 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00907594 s, 1.2 GB/s 2024-04-22T22:21:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-04-22T22:21:08.025 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:08 smithi113 ceph-mon[24083]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T22:21:08.049 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi113:vg_nvme/lv_3 2024-04-22T22:21:08.387 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:08 smithi047 ceph-mon[21846]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T22:21:09.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:09 smithi113 ceph-mon[24083]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:09.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:09 smithi113 ceph-mon[24083]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T22:21:09.241 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:09 smithi113 ceph-mon[24083]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T22:21:09.387 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:09 smithi047 ceph-mon[21846]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:09.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:09 smithi047 ceph-mon[21846]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T22:21:09.388 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:09 smithi047 ceph-mon[21846]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T22:21:10.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T22:21:10.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: osdmap e36: 5 total, 5 up, 5 in 2024-04-22T22:21:10.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: Detected new or changed devices on smithi113 2024-04-22T22:21:10.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:10.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:10.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: Adjusting osd_memory_target on smithi113 to 15936M 2024-04-22T22:21:10.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:10.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:10.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:10.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:10.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:10.639 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:10.740 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T22:21:10.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: osdmap e36: 5 total, 5 up, 5 in 2024-04-22T22:21:10.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: Detected new or changed devices on smithi113 2024-04-22T22:21:10.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:10.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:10.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: Adjusting osd_memory_target on smithi113 to 15936M 2024-04-22T22:21:10.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:10.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:10.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:10.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:10.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:10.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:11.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:11 smithi047 ceph-mon[21846]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:21:11.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:11 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:21:11.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:11 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:21:11.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:11 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:11.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:11 smithi113 ceph-mon[24083]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:21:11.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:11 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:21:11.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:11 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:21:11.738 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:11 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:12.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:12 smithi047 ceph-mon[21846]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-22T22:21:12.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:12 smithi113 ceph-mon[24083]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-22T22:21:13.637 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:13 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/635871857' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ba50386-62d7-4738-b5ae-6ed284df22ca"}]: dispatch 2024-04-22T22:21:13.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:13 smithi047 ceph-mon[21846]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ba50386-62d7-4738-b5ae-6ed284df22ca"}]: dispatch 2024-04-22T22:21:13.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:13 smithi047 ceph-mon[21846]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ba50386-62d7-4738-b5ae-6ed284df22ca"}]': finished 2024-04-22T22:21:13.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:13 smithi047 ceph-mon[21846]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T22:21:13.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:13 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:13.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:13 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/635871857' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ba50386-62d7-4738-b5ae-6ed284df22ca"}]: dispatch 2024-04-22T22:21:13.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:13 smithi113 ceph-mon[24083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ba50386-62d7-4738-b5ae-6ed284df22ca"}]: dispatch 2024-04-22T22:21:13.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:13 smithi113 ceph-mon[24083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ba50386-62d7-4738-b5ae-6ed284df22ca"}]': finished 2024-04-22T22:21:13.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:13 smithi113 ceph-mon[24083]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T22:21:13.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:13 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:14.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:14 smithi047 ceph-mon[21846]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-22T22:21:14.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:14 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/1960988192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:21:14.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:14 smithi113 ceph-mon[24083]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-22T22:21:14.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:14 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/1960988192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:21:15.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:21:15.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:21:15.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:21:15.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:21:16.571 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:16 smithi113 ceph-mon[24083]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:16.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:16 smithi047 ceph-mon[21846]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:18.491 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:18 smithi113 ceph-mon[24083]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:18.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:18 smithi047 ceph-mon[21846]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:19.504 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:19 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T22:21:19.504 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:19 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:19.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T22:21:19.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:20.688 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:20 smithi113 ceph-mon[24083]: Deploying daemon osd.5 on smithi113 2024-04-22T22:21:20.688 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:20 smithi113 ceph-mon[24083]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:20.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:20 smithi047 ceph-mon[21846]: Deploying daemon osd.5 on smithi113 2024-04-22T22:21:20.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:20 smithi047 ceph-mon[21846]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:22.658 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:22 smithi113 ceph-mon[24083]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:22.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:22 smithi047 ceph-mon[21846]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:24.641 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:24 smithi113 ceph-mon[24083]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:24.641 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:24 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:24.641 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:24 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:24.641 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:24 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:24.641 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:24 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:24.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:24 smithi047 ceph-mon[21846]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:24.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:24 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:24.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:24 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:24.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:24 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:24.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:24 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:26.492 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:26 smithi113 ceph-mon[24083]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:26.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:26 smithi047 ceph-mon[21846]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:27.655 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:27 smithi113 ceph-mon[24083]: from='osd.5 [v2:172.21.15.113:6808/2782033586,v1:172.21.15.113:6809/2782033586]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T22:21:27.656 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:27 smithi113 ceph-mon[24083]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T22:21:27.793 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 5 on host 'smithi113' 2024-04-22T22:21:27.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:27 smithi047 ceph-mon[21846]: from='osd.5 [v2:172.21.15.113:6808/2782033586,v1:172.21.15.113:6809/2782033586]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T22:21:27.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:27 smithi047 ceph-mon[21846]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T22:21:28.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:28 smithi113 ceph-mon[24083]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:28.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:28 smithi113 ceph-mon[24083]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T22:21:28.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:28 smithi113 ceph-mon[24083]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T22:21:28.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:28 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:28.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:28 smithi113 ceph-mon[24083]: from='osd.5 [v2:172.21.15.113:6808/2782033586,v1:172.21.15.113:6809/2782033586]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:28.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:28 smithi113 ceph-mon[24083]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:28.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:28 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:28.795 DEBUG:teuthology.orchestra.run.smithi113:osd.5> sudo journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T22:21:28.797 INFO:tasks.cephadm:Deploying osd.6 on smithi113 with /dev/vg_nvme/lv_2... 2024-04-22T22:21:28.798 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T22:21:28.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:28 smithi047 ceph-mon[21846]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:28.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:28 smithi047 ceph-mon[21846]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T22:21:28.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:28 smithi047 ceph-mon[21846]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T22:21:28.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:28.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:28 smithi047 ceph-mon[21846]: from='osd.5 [v2:172.21.15.113:6808/2782033586,v1:172.21.15.113:6809/2782033586]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:28.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:28 smithi047 ceph-mon[21846]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:28.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:28 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:29.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:29 smithi113 ceph-mon[24083]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-22T22:21:29.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:29 smithi113 ceph-mon[24083]: osdmap e39: 6 total, 5 up, 6 in 2024-04-22T22:21:29.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:29 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:29.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:29 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:29.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:29 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:29.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:29 smithi047 ceph-mon[21846]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-22T22:21:29.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:29 smithi047 ceph-mon[21846]: osdmap e39: 6 total, 5 up, 6 in 2024-04-22T22:21:29.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:29 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:29.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:29 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:29.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:29 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:30.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:30 smithi113 ceph-mon[24083]: purged_snaps scrub starts 2024-04-22T22:21:30.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:30 smithi113 ceph-mon[24083]: purged_snaps scrub ok 2024-04-22T22:21:30.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:30 smithi113 ceph-mon[24083]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:30.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:30 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:30.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:30 smithi113 ceph-mon[24083]: osd.5 [v2:172.21.15.113:6808/2782033586,v1:172.21.15.113:6809/2782033586] boot 2024-04-22T22:21:30.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:30 smithi113 ceph-mon[24083]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T22:21:30.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:30 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:30.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:30 smithi047 ceph-mon[21846]: purged_snaps scrub starts 2024-04-22T22:21:30.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:30 smithi047 ceph-mon[21846]: purged_snaps scrub ok 2024-04-22T22:21:30.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:30 smithi047 ceph-mon[21846]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:21:30.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:30 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:30.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:30 smithi047 ceph-mon[21846]: osd.5 [v2:172.21.15.113:6808/2782033586,v1:172.21.15.113:6809/2782033586] boot 2024-04-22T22:21:30.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:30 smithi047 ceph-mon[21846]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T22:21:30.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:30 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:21:31.572 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:31 smithi113 ceph-mon[24083]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T22:21:31.638 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:31 smithi047 ceph-mon[21846]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T22:21:32.133 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T22:21:32.134 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T22:21:32.134 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-04-22T22:21:32.134 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-04-22T22:21:32.134 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0102414 s, 1.0 GB/s 2024-04-22T22:21:32.134 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-04-22T22:21:32.715 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:32 smithi113 ceph-mon[24083]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:32.715 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:32 smithi113 ceph-mon[24083]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-22T22:21:32.715 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:32 smithi113 ceph-mon[24083]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T22:21:32.740 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi113:vg_nvme/lv_2 2024-04-22T22:21:32.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:32 smithi047 ceph-mon[21846]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:32.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:32 smithi047 ceph-mon[21846]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-22T22:21:32.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:32 smithi047 ceph-mon[21846]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T22:21:33.665 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:33 smithi113 ceph-mon[24083]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T22:21:33.665 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:33 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:33.665 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:33 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:33.665 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:33 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:33.665 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:33 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:33.665 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:33 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:33.665 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:33 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:33.665 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:33 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:33.666 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:33 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:33.666 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:33 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:33.666 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:33 smithi113 ceph-mon[24083]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T22:21:33.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:33 smithi047 ceph-mon[21846]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T22:21:33.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:33 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:33.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:33 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:33.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:33 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:33.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:33 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:33.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:33 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:33.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:33 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:33.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:33 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:33.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:33 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:33.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:33 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:33.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:33 smithi047 ceph-mon[21846]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T22:21:34.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:34 smithi113 ceph-mon[24083]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:34.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:34 smithi113 ceph-mon[24083]: Detected new or changed devices on smithi113 2024-04-22T22:21:34.737 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:34 smithi113 ceph-mon[24083]: Adjusting osd_memory_target on smithi113 to 7968M 2024-04-22T22:21:34.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:34 smithi047 ceph-mon[21846]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:34.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:34 smithi047 ceph-mon[21846]: Detected new or changed devices on smithi113 2024-04-22T22:21:34.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:34 smithi047 ceph-mon[21846]: Adjusting osd_memory_target on smithi113 to 7968M 2024-04-22T22:21:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:35 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:21:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:35 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:21:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:35 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:35.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:35 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:21:35.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:35 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:21:35.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:35 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:36.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:36 smithi113 ceph-mon[24083]: from='client.24183 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:21:36.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:36 smithi113 ceph-mon[24083]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:36.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:36 smithi047 ceph-mon[21846]: from='client.24183 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:21:36.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:36 smithi047 ceph-mon[21846]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:37.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:37 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/1664846058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4e949a7-ba08-4503-adaa-6a987554323f"}]: dispatch 2024-04-22T22:21:37.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:37 smithi113 ceph-mon[24083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4e949a7-ba08-4503-adaa-6a987554323f"}]: dispatch 2024-04-22T22:21:37.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:37 smithi113 ceph-mon[24083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4e949a7-ba08-4503-adaa-6a987554323f"}]': finished 2024-04-22T22:21:37.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:37 smithi113 ceph-mon[24083]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T22:21:37.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:21:37.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:37 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/1664846058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4e949a7-ba08-4503-adaa-6a987554323f"}]: dispatch 2024-04-22T22:21:37.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:37 smithi047 ceph-mon[21846]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4e949a7-ba08-4503-adaa-6a987554323f"}]: dispatch 2024-04-22T22:21:37.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:37 smithi047 ceph-mon[21846]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4e949a7-ba08-4503-adaa-6a987554323f"}]': finished 2024-04-22T22:21:37.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:37 smithi047 ceph-mon[21846]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T22:21:37.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:21:38.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:38 smithi113 ceph-mon[24083]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T22:21:38.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:38 smithi113 ceph-mon[24083]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-22T22:21:38.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:38 smithi113 ceph-mon[24083]: Cluster is now healthy 2024-04-22T22:21:38.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:38 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/2547150631' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:21:38.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:38 smithi047 ceph-mon[21846]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T22:21:38.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:38 smithi047 ceph-mon[21846]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-22T22:21:38.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:38 smithi047 ceph-mon[21846]: Cluster is now healthy 2024-04-22T22:21:38.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:38 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/2547150631' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:21:40.492 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:40 smithi113 ceph-mon[24083]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-22T22:21:40.888 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:40 smithi047 ceph-mon[21846]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-22T22:21:42.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:42 smithi113 ceph-mon[24083]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-22T22:21:42.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:42 smithi047 ceph-mon[21846]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-22T22:21:44.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:44 smithi113 ceph-mon[24083]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T22:21:44.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:44 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T22:21:44.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:44 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:44.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:44 smithi113 ceph-mon[24083]: Deploying daemon osd.6 on smithi113 2024-04-22T22:21:44.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:44 smithi047 ceph-mon[21846]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T22:21:44.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:44 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T22:21:44.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:44 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:44.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:44 smithi047 ceph-mon[21846]: Deploying daemon osd.6 on smithi113 2024-04-22T22:21:46.648 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:46 smithi113 ceph-mon[24083]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T22:21:46.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:46 smithi047 ceph-mon[21846]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T22:21:48.592 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:48 smithi113 ceph-mon[24083]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:48.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:48 smithi047 ceph-mon[21846]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:49.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:49.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:49.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:49.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:49 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:49.889 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:49.890 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:49.890 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:49.890 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:49 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:50.855 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:50 smithi113 ceph-mon[24083]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:50.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:50 smithi047 ceph-mon[21846]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:52.412 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 6 on host 'smithi113' 2024-04-22T22:21:52.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:52 smithi113 ceph-mon[24083]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:52.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:52 smithi113 ceph-mon[24083]: from='osd.6 [v2:172.21.15.113:6816/3350371263,v1:172.21.15.113:6817/3350371263]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T22:21:52.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:52 smithi113 ceph-mon[24083]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T22:21:52.741 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:52 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:52.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:52 smithi047 ceph-mon[21846]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:52.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:52 smithi047 ceph-mon[21846]: from='osd.6 [v2:172.21.15.113:6816/3350371263,v1:172.21.15.113:6817/3350371263]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T22:21:52.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:52 smithi047 ceph-mon[21846]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T22:21:52.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:52 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:53.090 DEBUG:teuthology.orchestra.run.smithi113:osd.6> sudo journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.6.service 2024-04-22T22:21:53.093 INFO:tasks.cephadm:Deploying osd.7 on smithi113 with /dev/vg_nvme/lv_1... 2024-04-22T22:21:53.093 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T22:21:53.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:53 smithi047 ceph-mon[21846]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T22:21:53.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:53 smithi047 ceph-mon[21846]: osdmap e46: 7 total, 6 up, 7 in 2024-04-22T22:21:53.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:53 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:21:53.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:53 smithi047 ceph-mon[21846]: from='osd.6 [v2:172.21.15.113:6816/3350371263,v1:172.21.15.113:6817/3350371263]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:53.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:53 smithi047 ceph-mon[21846]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:53.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:53 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:53.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:53 smithi113 ceph-mon[24083]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T22:21:53.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:53 smithi113 ceph-mon[24083]: osdmap e46: 7 total, 6 up, 7 in 2024-04-22T22:21:53.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:53 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:21:53.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:53 smithi113 ceph-mon[24083]: from='osd.6 [v2:172.21.15.113:6816/3350371263,v1:172.21.15.113:6817/3350371263]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:53.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:53 smithi113 ceph-mon[24083]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:21:53.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:53 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:53.993 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:21:53 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:21:53.605+0000 7fc6f218b700 -1 osd.6 0 waiting for initial osdmap 2024-04-22T22:21:53.993 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:21:53 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:21:53.612+0000 7fc6eab1e700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:21:54.612 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:54 smithi113 ceph-mon[24083]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:54.612 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:54 smithi113 ceph-mon[24083]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-22T22:21:54.612 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:54 smithi113 ceph-mon[24083]: osdmap e47: 7 total, 6 up, 7 in 2024-04-22T22:21:54.612 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:54 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:21:54.612 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:54 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:21:54.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:54 smithi047 ceph-mon[21846]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:54.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:54 smithi047 ceph-mon[21846]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-22T22:21:54.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:54 smithi047 ceph-mon[21846]: osdmap e47: 7 total, 6 up, 7 in 2024-04-22T22:21:54.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:54 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:21:54.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:54 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:21:55.119 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T22:21:55.119 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T22:21:55.119 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-04-22T22:21:55.119 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-04-22T22:21:55.119 INFO:teuthology.orchestra.run.smithi113.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0103636 s, 1.0 GB/s 2024-04-22T22:21:55.119 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 2024-04-22T22:21:55.119 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-04-22T22:21:55.742 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:55 smithi113 ceph-mon[24083]: purged_snaps scrub starts 2024-04-22T22:21:55.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:55 smithi113 ceph-mon[24083]: purged_snaps scrub ok 2024-04-22T22:21:55.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:55 smithi113 ceph-mon[24083]: osd.6 [v2:172.21.15.113:6816/3350371263,v1:172.21.15.113:6817/3350371263] boot 2024-04-22T22:21:55.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:55 smithi113 ceph-mon[24083]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T22:21:55.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:55 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:21:55.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:55 smithi047 ceph-mon[21846]: purged_snaps scrub starts 2024-04-22T22:21:55.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:55 smithi047 ceph-mon[21846]: purged_snaps scrub ok 2024-04-22T22:21:55.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:55 smithi047 ceph-mon[21846]: osd.6 [v2:172.21.15.113:6816/3350371263,v1:172.21.15.113:6817/3350371263] boot 2024-04-22T22:21:55.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:55 smithi047 ceph-mon[21846]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T22:21:55.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:55 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:21:56.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:56 smithi113 ceph-mon[24083]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:56.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:56 smithi113 ceph-mon[24083]: osdmap e49: 7 total, 7 up, 7 in 2024-04-22T22:21:56.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:56 smithi047 ceph-mon[21846]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:21:56.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:56 smithi047 ceph-mon[21846]: osdmap e49: 7 total, 7 up, 7 in 2024-04-22T22:21:57.155 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi113:vg_nvme/lv_1 2024-04-22T22:21:57.887 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:57 smithi113 ceph-mon[24083]: osdmap e50: 7 total, 7 up, 7 in 2024-04-22T22:21:57.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:57 smithi047 ceph-mon[21846]: osdmap e50: 7 total, 7 up, 7 in 2024-04-22T22:21:58.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:21:58.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: osdmap e51: 7 total, 7 up, 7 in 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: Detected new or changed devices on smithi113 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: Adjusting osd_memory_target on smithi113 to 5312M 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:58.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:58.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:21:58.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: osdmap e51: 7 total, 7 up, 7 in 2024-04-22T22:21:58.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: Detected new or changed devices on smithi113 2024-04-22T22:21:58.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:58.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:58.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:58.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:21:58.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: Adjusting osd_memory_target on smithi113 to 5312M 2024-04-22T22:21:58.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:58.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:58.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:21:58.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:21:58.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:21:58.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:21:59.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:21:59 smithi047 ceph-mon[21846]: osdmap e52: 7 total, 7 up, 7 in 2024-04-22T22:21:59.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:21:59 smithi113 ceph-mon[24083]: osdmap e52: 7 total, 7 up, 7 in 2024-04-22T22:22:00.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:00 smithi113 ceph-mon[24083]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:00.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:00 smithi113 ceph-mon[24083]: from='client.24201 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:22:00.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:22:00.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:22:00.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:00 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:22:00.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:00 smithi047 ceph-mon[21846]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:00.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:00 smithi047 ceph-mon[21846]: from='client.24201 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:22:00.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:22:00.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:22:00.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:00 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:22:02.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:02 smithi047 ceph-mon[21846]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-22T22:22:02.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:02 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/1540500339' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e5c240a-9909-4c1c-bdf4-a527f84170b0"}]: dispatch 2024-04-22T22:22:02.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:02 smithi047 ceph-mon[21846]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e5c240a-9909-4c1c-bdf4-a527f84170b0"}]: dispatch 2024-04-22T22:22:02.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:02 smithi047 ceph-mon[21846]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e5c240a-9909-4c1c-bdf4-a527f84170b0"}]': finished 2024-04-22T22:22:02.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:02 smithi047 ceph-mon[21846]: osdmap e53: 8 total, 7 up, 8 in 2024-04-22T22:22:02.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:02.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:02 smithi113 ceph-mon[24083]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-22T22:22:02.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:02 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/1540500339' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e5c240a-9909-4c1c-bdf4-a527f84170b0"}]: dispatch 2024-04-22T22:22:02.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:02 smithi113 ceph-mon[24083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e5c240a-9909-4c1c-bdf4-a527f84170b0"}]: dispatch 2024-04-22T22:22:02.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:02 smithi113 ceph-mon[24083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e5c240a-9909-4c1c-bdf4-a527f84170b0"}]': finished 2024-04-22T22:22:02.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:02 smithi113 ceph-mon[24083]: osdmap e53: 8 total, 7 up, 8 in 2024-04-22T22:22:02.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:03.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:03 smithi113 ceph-mon[24083]: from='client.? 172.21.15.113:0/623203844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:22:03.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:03 smithi047 ceph-mon[21846]: from='client.? 172.21.15.113:0/623203844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:22:04.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:04 smithi047 ceph-mon[21846]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T22:22:04.905 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:04 smithi113 ceph-mon[24083]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T22:22:06.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:06 smithi047 ceph-mon[21846]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-22T22:22:06.953 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:06 smithi113 ceph-mon[24083]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-22T22:22:08.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:08 smithi113 ceph-mon[24083]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-22T22:22:08.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:08 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T22:22:08.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:08 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:22:08.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:08 smithi047 ceph-mon[21846]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-22T22:22:08.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:08 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T22:22:08.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:08 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:22:09.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:09 smithi047 ceph-mon[21846]: Deploying daemon osd.7 on smithi113 2024-04-22T22:22:09.975 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:09 smithi113 ceph-mon[24083]: Deploying daemon osd.7 on smithi113 2024-04-22T22:22:10.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:10 smithi047 ceph-mon[21846]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T22:22:10.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:10 smithi113 ceph-mon[24083]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T22:22:12.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:12 smithi113 ceph-mon[24083]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:12.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:12 smithi047 ceph-mon[21846]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:14.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:14 smithi047 ceph-mon[21846]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:14.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:14 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:14.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:14 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:22:14.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:14 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:22:14.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:14 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:22:14.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:14 smithi113 ceph-mon[24083]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:14.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:14 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:14.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:14 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:22:14.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:14 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:22:14.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:14 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:22:15.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:22:15.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:22:15.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:22:15.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:22:16.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:16 smithi113 ceph-mon[24083]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:16.794 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 7 on host 'smithi113' 2024-04-22T22:22:16.889 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:16 smithi047 ceph-mon[21846]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:17.499 DEBUG:teuthology.orchestra.run.smithi113:osd.7> sudo journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.7.service 2024-04-22T22:22:17.502 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-22T22:22:17.502 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T22:22:17.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:17 smithi113 ceph-mon[24083]: from='osd.7 [v2:172.21.15.113:6824/2506384215,v1:172.21.15.113:6825/2506384215]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T22:22:17.743 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:17 smithi113 ceph-mon[24083]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T22:22:17.744 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:17 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:17.823 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:17 smithi047 ceph-mon[21846]: from='osd.7 [v2:172.21.15.113:6824/2506384215,v1:172.21.15.113:6825/2506384215]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T22:22:17.823 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:17 smithi047 ceph-mon[21846]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T22:22:17.824 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:17 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:18.744 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:18 smithi113 ceph-mon[24083]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:18.744 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:18 smithi113 ceph-mon[24083]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T22:22:18.744 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:18 smithi113 ceph-mon[24083]: osdmap e54: 8 total, 7 up, 8 in 2024-04-22T22:22:18.744 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:18 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:18.744 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:18 smithi113 ceph-mon[24083]: from='osd.7 [v2:172.21.15.113:6824/2506384215,v1:172.21.15.113:6825/2506384215]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:22:18.744 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:18 smithi113 ceph-mon[24083]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:22:18.744 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:18 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:18.744 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:22:18 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:22:18.666+0000 7f4f9e1bf700 -1 osd.7 0 waiting for initial osdmap 2024-04-22T22:22:18.744 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:22:18 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:22:18.673+0000 7f4f98b56700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:22:18.954 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:18 smithi047 ceph-mon[21846]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:18.954 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:18 smithi047 ceph-mon[21846]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T22:22:18.954 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:18 smithi047 ceph-mon[21846]: osdmap e54: 8 total, 7 up, 8 in 2024-04-22T22:22:18.954 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:18 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:18.954 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:18 smithi047 ceph-mon[21846]: from='osd.7 [v2:172.21.15.113:6824/2506384215,v1:172.21.15.113:6825/2506384215]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:22:18.954 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:18 smithi047 ceph-mon[21846]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-22T22:22:18.954 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:18 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:19.710 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:22:19.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:19 smithi113 ceph-mon[24083]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-22T22:22:19.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:19 smithi113 ceph-mon[24083]: osdmap e55: 8 total, 7 up, 8 in 2024-04-22T22:22:19.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:19 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:19.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:19 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:20.072 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:19 smithi047 ceph-mon[21846]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-22T22:22:20.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:19 smithi047 ceph-mon[21846]: osdmap e55: 8 total, 7 up, 8 in 2024-04-22T22:22:20.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:20.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:19 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:20.386 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1713824539,"num_in_osds":8,"osd_in_since":1713824522,"num_remapped_pgs":1} 2024-04-22T22:22:20.387 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T22:22:20.632 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:20.969 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:20 smithi047 ceph-mon[21846]: purged_snaps scrub starts 2024-04-22T22:22:20.969 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:20 smithi047 ceph-mon[21846]: purged_snaps scrub ok 2024-04-22T22:22:20.969 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:20 smithi047 ceph-mon[21846]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:20.969 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:20.969 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:20 smithi047 ceph-mon[21846]: osd.7 [v2:172.21.15.113:6824/2506384215,v1:172.21.15.113:6825/2506384215] boot 2024-04-22T22:22:20.969 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:20 smithi047 ceph-mon[21846]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T22:22:20.969 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:20 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:20.969 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:20 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2545809013' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T22:22:20.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:20 smithi113 ceph-mon[24083]: purged_snaps scrub starts 2024-04-22T22:22:20.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:20 smithi113 ceph-mon[24083]: purged_snaps scrub ok 2024-04-22T22:22:20.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:20 smithi113 ceph-mon[24083]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:22:20.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:20 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:20.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:20 smithi113 ceph-mon[24083]: osd.7 [v2:172.21.15.113:6824/2506384215,v1:172.21.15.113:6825/2506384215] boot 2024-04-22T22:22:20.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:20 smithi113 ceph-mon[24083]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T22:22:20.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:20 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:22:20.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:20 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2545809013' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T22:22:21.732 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T22:22:21.732 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:21.732 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:22:21.732 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:22:21.732 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:22:21.732 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:22:21.733 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:21.733 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:21.733 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:22:21.733 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:22:21.733 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:22:21.733 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:21.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T22:22:21.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:21.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:22:21.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:22:21.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:22:21.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:22:21.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:21.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:21.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:22:21.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:22:21.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:22:21.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:22:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:22:22.444 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":58,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","created":"2024-04-22T22:16:23.411932+0000","modified":"2024-04-22T22:22:21.671338+0000","last_up_change":"2024-04-22T22:22:19.663178+0000","last_in_change":"2024-04-22T22:22:02.071260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T22:20:17.786154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"eaaaa67b-8d25-4a21-ad4f-299f01bbaa73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6803","nonce":2475417589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6805","nonce":2475417589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6809","nonce":2475417589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6807","nonce":2475417589}]},"public_addr":"172.21.15.47:6803/2475417589","cluster_addr":"172.21.15.47:6805/2475417589","heartbeat_back_addr":"172.21.15.47:6809/2475417589","heartbeat_front_addr":"172.21.15.47:6807/2475417589","state":["exists","up"]},{"osd":1,"uuid":"96a3a82d-1779-4fe6-96da-9071bbd29214","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6811","nonce":3406095492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6813","nonce":3406095492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6817","nonce":3406095492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6815","nonce":3406095492}]},"public_addr":"172.21.15.47:6811/3406095492","cluster_addr":"172.21.15.47:6813/3406095492","heartbeat_back_addr":"172.21.15.47:6817/3406095492","heartbeat_front_addr":"172.21.15.47:6815/3406095492","state":["exists","up"]},{"osd":2,"uuid":"0de44f6b-8fea-441f-9de7-dc6d1156164e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6819","nonce":2629230070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6821","nonce":2629230070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6825","nonce":2629230070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6823","nonce":2629230070}]},"public_addr":"172.21.15.47:6819/2629230070","cluster_addr":"172.21.15.47:6821/2629230070","heartbeat_back_addr":"172.21.15.47:6825/2629230070","heartbeat_front_addr":"172.21.15.47:6823/2629230070","state":["exists","up"]},{"osd":3,"uuid":"d0eff041-a458-45ed-ae2e-288d257c7d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6827","nonce":1765039522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6829","nonce":1765039522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6833","nonce":1765039522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6831","nonce":1765039522}]},"public_addr":"172.21.15.47:6827/1765039522","cluster_addr":"172.21.15.47:6829/1765039522","heartbeat_back_addr":"172.21.15.47:6833/1765039522","heartbeat_front_addr":"172.21.15.47:6831/1765039522","state":["exists","up"]},{"osd":4,"uuid":"3f41cc9d-8c00-45b5-a9a6-b86071e2a081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6801","nonce":2948692086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6803","nonce":2948692086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6807","nonce":2948692086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6805","nonce":2948692086}]},"public_addr":"172.21.15.113:6801/2948692086","cluster_addr":"172.21.15.113:6803/2948692086","heartbeat_back_addr":"172.21.15.113:6807/2948692086","heartbeat_front_addr":"172.21.15.113:6805/2948692086","state":["exists","up"]},{"osd":5,"uuid":"1ba50386-62d7-4738-b5ae-6ed284df22ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6809","nonce":2782033586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6811","nonce":2782033586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6815","nonce":2782033586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6813","nonce":2782033586}]},"public_addr":"172.21.15.113:6809/2782033586","cluster_addr":"172.21.15.113:6811/2782033586","heartbeat_back_addr":"172.21.15.113:6815/2782033586","heartbeat_front_addr":"172.21.15.113:6813/2782033586","state":["exists","up"]},{"osd":6,"uuid":"a4e949a7-ba08-4503-adaa-6a987554323f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6817","nonce":3350371263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6819","nonce":3350371263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6823","nonce":3350371263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6821","nonce":3350371263}]},"public_addr":"172.21.15.113:6817/3350371263","cluster_addr":"172.21.15.113:6819/3350371263","heartbeat_back_addr":"172.21.15.113:6823/3350371263","heartbeat_front_addr":"172.21.15.113:6821/3350371263","state":["exists","up"]},{"osd":7,"uuid":"1e5c240a-9909-4c1c-bdf4-a527f84170b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6825","nonce":2506384215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6827","nonce":2506384215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6831","nonce":2506384215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6829","nonce":2506384215}]},"public_addr":"172.21.15.113:6825/2506384215","cluster_addr":"172.21.15.113:6827/2506384215","heartbeat_back_addr":"172.21.15.113:6831/2506384215","heartbeat_front_addr":"172.21.15.113:6829/2506384215","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:19:18.547544+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:19:45.487311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:20:13.883339+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:20:42.284248+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:21:04.054098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:21:28.385452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:21:52.670320+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:22:17.791344+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[0,6]}],"primary_temp":[],"blocklist":{"172.21.15.47:6801/293375230":"2024-04-23T22:18:15.224730+0000","172.21.15.47:6800/293375230":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/3586612862":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/869020271":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/2868740181":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/896685020":"2024-04-23T22:17:19.978457+0000","172.21.15.47:0/1128111817":"2024-04-23T22:17:19.978457+0000","172.21.15.47:6800/2320765223":"2024-04-23T22:17:19.978457+0000","172.21.15.47:0/1250921770":"2024-04-23T22:16:50.356523+0000","172.21.15.47:6801/2320765223":"2024-04-23T22:17:19.978457+0000","172.21.15.47:6800/680675122":"2024-04-23T22:16:50.356523+0000","172.21.15.47:0/1207367291":"2024-04-23T22:17:19.978457+0000","172.21.15.47:6801/680675122":"2024-04-23T22:16:50.356523+0000","172.21.15.47:0/3158220163":"2024-04-23T22:16:50.356523+0000","172.21.15.47:0/3814644933":"2024-04-23T22:16:50.356523+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T22:22:22.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:22 smithi113 ceph-mon[24083]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:22.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:22 smithi113 ceph-mon[24083]: Detected new or changed devices on smithi113 2024-04-22T22:22:22.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:22 smithi113 ceph-mon[24083]: Adjusting osd_memory_target on smithi113 to 3984M 2024-04-22T22:22:22.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:22 smithi113 ceph-mon[24083]: osdmap e58: 8 total, 8 up, 8 in 2024-04-22T22:22:22.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:22 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2472387530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:22:23.140 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:22 smithi047 ceph-mon[21846]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:23.140 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:22 smithi047 ceph-mon[21846]: Detected new or changed devices on smithi113 2024-04-22T22:22:23.140 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:22 smithi047 ceph-mon[21846]: Adjusting osd_memory_target on smithi113 to 3984M 2024-04-22T22:22:23.140 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:22 smithi047 ceph-mon[21846]: osdmap e58: 8 total, 8 up, 8 in 2024-04-22T22:22:23.140 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:22 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2472387530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:22:23.193 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T22:20:17.786154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-22T22:22:23.193 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T22:22:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:23.874 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:23 smithi047 ceph-mon[21846]: osdmap e59: 8 total, 8 up, 8 in 2024-04-22T22:22:23.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:23 smithi113 ceph-mon[24083]: osdmap e59: 8 total, 8 up, 8 in 2024-04-22T22:22:24.874 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:24 smithi047 ceph-mon[21846]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:24.874 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:24 smithi047 ceph-mon[21846]: osdmap e60: 8 total, 8 up, 8 in 2024-04-22T22:22:24.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:24 smithi113 ceph-mon[24083]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:24.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:24 smithi113 ceph-mon[24083]: osdmap e60: 8 total, 8 up, 8 in 2024-04-22T22:22:25.284 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-04-22T22:22:25.993 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:25 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/1856277433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T22:22:25.999 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:25 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1856277433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T22:22:26.026 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T22:22:26.027 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T22:22:26.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:26 smithi113 ceph-mon[24083]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:27.010 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:26 smithi047 ceph-mon[21846]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:28.171 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-04-22T22:22:28.171 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAk4yZmborOChAAGCAYuDVK+J9oHZXwb5tc6w== 2024-04-22T22:22:28.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:28 smithi047 ceph-mon[21846]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T22:22:28.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:28 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2628261748' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:22:28.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:28 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2628261748' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T22:22:28.920 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-22T22:22:28.920 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T22:22:28.920 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T22:22:28.958 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T22:22:28.958 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T22:22:28.959 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T22:22:28.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:28 smithi113 ceph-mon[24083]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T22:22:28.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:28 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2628261748' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:22:28.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:28 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2628261748' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T22:22:29.217 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:30.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:30 smithi047 ceph-mon[21846]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-22T22:22:30.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:30 smithi113 ceph-mon[24083]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-22T22:22:31.086 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:22:31.733 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":19,"active_gid":14206,"active_name":"smithi047.uutpnj","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":4068623798},{"type":"v1","addr":"172.21.15.47:6801","nonce":4068623798}]},"active_addr":"172.21.15.47:6801/4068623798","active_change":"2024-04-22T22:18:15.225082+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14224,"name":"smithi113.qvxssq","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.47:8443/","prometheus":"http://172.21.15.47:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":255298117}]},{"addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":1615917728}]},{"addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":3931157397}]},{"addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":1215953760}]}]}} 2024-04-22T22:22:31.739 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T22:22:31.739 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T22:22:31.739 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T22:22:31.959 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:31.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:31 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/1327566388' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T22:22:32.075 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:31 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1327566388' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T22:22:32.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:32 smithi113 ceph-mon[24083]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-22T22:22:33.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:32 smithi047 ceph-mon[21846]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-22T22:22:33.813 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:22:33.813 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":60,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","created":"2024-04-22T22:16:23.411932+0000","modified":"2024-04-22T22:22:23.681418+0000","last_up_change":"2024-04-22T22:22:19.663178+0000","last_in_change":"2024-04-22T22:22:02.071260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T22:20:17.786154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"eaaaa67b-8d25-4a21-ad4f-299f01bbaa73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6803","nonce":2475417589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6805","nonce":2475417589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6809","nonce":2475417589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6807","nonce":2475417589}]},"public_addr":"172.21.15.47:6803/2475417589","cluster_addr":"172.21.15.47:6805/2475417589","heartbeat_back_addr":"172.21.15.47:6809/2475417589","heartbeat_front_addr":"172.21.15.47:6807/2475417589","state":["exists","up"]},{"osd":1,"uuid":"96a3a82d-1779-4fe6-96da-9071bbd29214","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6811","nonce":3406095492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6813","nonce":3406095492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6817","nonce":3406095492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6815","nonce":3406095492}]},"public_addr":"172.21.15.47:6811/3406095492","cluster_addr":"172.21.15.47:6813/3406095492","heartbeat_back_addr":"172.21.15.47:6817/3406095492","heartbeat_front_addr":"172.21.15.47:6815/3406095492","state":["exists","up"]},{"osd":2,"uuid":"0de44f6b-8fea-441f-9de7-dc6d1156164e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6819","nonce":2629230070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6821","nonce":2629230070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6825","nonce":2629230070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6823","nonce":2629230070}]},"public_addr":"172.21.15.47:6819/2629230070","cluster_addr":"172.21.15.47:6821/2629230070","heartbeat_back_addr":"172.21.15.47:6825/2629230070","heartbeat_front_addr":"172.21.15.47:6823/2629230070","state":["exists","up"]},{"osd":3,"uuid":"d0eff041-a458-45ed-ae2e-288d257c7d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6827","nonce":1765039522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6829","nonce":1765039522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6833","nonce":1765039522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6831","nonce":1765039522}]},"public_addr":"172.21.15.47:6827/1765039522","cluster_addr":"172.21.15.47:6829/1765039522","heartbeat_back_addr":"172.21.15.47:6833/1765039522","heartbeat_front_addr":"172.21.15.47:6831/1765039522","state":["exists","up"]},{"osd":4,"uuid":"3f41cc9d-8c00-45b5-a9a6-b86071e2a081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6801","nonce":2948692086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6803","nonce":2948692086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6807","nonce":2948692086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6805","nonce":2948692086}]},"public_addr":"172.21.15.113:6801/2948692086","cluster_addr":"172.21.15.113:6803/2948692086","heartbeat_back_addr":"172.21.15.113:6807/2948692086","heartbeat_front_addr":"172.21.15.113:6805/2948692086","state":["exists","up"]},{"osd":5,"uuid":"1ba50386-62d7-4738-b5ae-6ed284df22ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6809","nonce":2782033586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6811","nonce":2782033586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6815","nonce":2782033586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6813","nonce":2782033586}]},"public_addr":"172.21.15.113:6809/2782033586","cluster_addr":"172.21.15.113:6811/2782033586","heartbeat_back_addr":"172.21.15.113:6815/2782033586","heartbeat_front_addr":"172.21.15.113:6813/2782033586","state":["exists","up"]},{"osd":6,"uuid":"a4e949a7-ba08-4503-adaa-6a987554323f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6817","nonce":3350371263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6819","nonce":3350371263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6823","nonce":3350371263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6821","nonce":3350371263}]},"public_addr":"172.21.15.113:6817/3350371263","cluster_addr":"172.21.15.113:6819/3350371263","heartbeat_back_addr":"172.21.15.113:6823/3350371263","heartbeat_front_addr":"172.21.15.113:6821/3350371263","state":["exists","up"]},{"osd":7,"uuid":"1e5c240a-9909-4c1c-bdf4-a527f84170b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6825","nonce":2506384215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6827","nonce":2506384215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6831","nonce":2506384215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6829","nonce":2506384215}]},"public_addr":"172.21.15.113:6825/2506384215","cluster_addr":"172.21.15.113:6827/2506384215","heartbeat_back_addr":"172.21.15.113:6831/2506384215","heartbeat_front_addr":"172.21.15.113:6829/2506384215","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:19:18.547544+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:19:45.487311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:20:13.883339+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:20:42.284248+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:21:04.054098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:21:28.385452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:21:52.670320+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:22:17.791344+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6801/293375230":"2024-04-23T22:18:15.224730+0000","172.21.15.47:6800/293375230":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/3586612862":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/869020271":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/2868740181":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/896685020":"2024-04-23T22:17:19.978457+0000","172.21.15.47:0/1128111817":"2024-04-23T22:17:19.978457+0000","172.21.15.47:6800/2320765223":"2024-04-23T22:17:19.978457+0000","172.21.15.47:0/1250921770":"2024-04-23T22:16:50.356523+0000","172.21.15.47:6801/2320765223":"2024-04-23T22:17:19.978457+0000","172.21.15.47:6800/680675122":"2024-04-23T22:16:50.356523+0000","172.21.15.47:0/1207367291":"2024-04-23T22:17:19.978457+0000","172.21.15.47:6801/680675122":"2024-04-23T22:16:50.356523+0000","172.21.15.47:0/3158220163":"2024-04-23T22:16:50.356523+0000","172.21.15.47:0/3814644933":"2024-04-23T22:16:50.356523+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T22:22:34.501 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T22:22:34.502 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T22:22:34.727 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:34.840 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:34 smithi047 ceph-mon[21846]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T22:22:34.840 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:34 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/993448916' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:22:34.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:34 smithi113 ceph-mon[24083]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T22:22:34.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:34 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/993448916' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:22:36.548 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:22:36.548 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":60,"fsid":"dbcf7660-00f5-11ef-bc93-c7b262605968","created":"2024-04-22T22:16:23.411932+0000","modified":"2024-04-22T22:22:23.681418+0000","last_up_change":"2024-04-22T22:22:19.663178+0000","last_in_change":"2024-04-22T22:22:02.071260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T22:20:17.786154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"eaaaa67b-8d25-4a21-ad4f-299f01bbaa73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6803","nonce":2475417589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6805","nonce":2475417589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6809","nonce":2475417589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2475417589},{"type":"v1","addr":"172.21.15.47:6807","nonce":2475417589}]},"public_addr":"172.21.15.47:6803/2475417589","cluster_addr":"172.21.15.47:6805/2475417589","heartbeat_back_addr":"172.21.15.47:6809/2475417589","heartbeat_front_addr":"172.21.15.47:6807/2475417589","state":["exists","up"]},{"osd":1,"uuid":"96a3a82d-1779-4fe6-96da-9071bbd29214","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6811","nonce":3406095492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6813","nonce":3406095492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6817","nonce":3406095492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":3406095492},{"type":"v1","addr":"172.21.15.47:6815","nonce":3406095492}]},"public_addr":"172.21.15.47:6811/3406095492","cluster_addr":"172.21.15.47:6813/3406095492","heartbeat_back_addr":"172.21.15.47:6817/3406095492","heartbeat_front_addr":"172.21.15.47:6815/3406095492","state":["exists","up"]},{"osd":2,"uuid":"0de44f6b-8fea-441f-9de7-dc6d1156164e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6819","nonce":2629230070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6821","nonce":2629230070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6825","nonce":2629230070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2629230070},{"type":"v1","addr":"172.21.15.47:6823","nonce":2629230070}]},"public_addr":"172.21.15.47:6819/2629230070","cluster_addr":"172.21.15.47:6821/2629230070","heartbeat_back_addr":"172.21.15.47:6825/2629230070","heartbeat_front_addr":"172.21.15.47:6823/2629230070","state":["exists","up"]},{"osd":3,"uuid":"d0eff041-a458-45ed-ae2e-288d257c7d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6827","nonce":1765039522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6829","nonce":1765039522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6833","nonce":1765039522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":1765039522},{"type":"v1","addr":"172.21.15.47:6831","nonce":1765039522}]},"public_addr":"172.21.15.47:6827/1765039522","cluster_addr":"172.21.15.47:6829/1765039522","heartbeat_back_addr":"172.21.15.47:6833/1765039522","heartbeat_front_addr":"172.21.15.47:6831/1765039522","state":["exists","up"]},{"osd":4,"uuid":"3f41cc9d-8c00-45b5-a9a6-b86071e2a081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6801","nonce":2948692086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6803","nonce":2948692086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6807","nonce":2948692086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2948692086},{"type":"v1","addr":"172.21.15.113:6805","nonce":2948692086}]},"public_addr":"172.21.15.113:6801/2948692086","cluster_addr":"172.21.15.113:6803/2948692086","heartbeat_back_addr":"172.21.15.113:6807/2948692086","heartbeat_front_addr":"172.21.15.113:6805/2948692086","state":["exists","up"]},{"osd":5,"uuid":"1ba50386-62d7-4738-b5ae-6ed284df22ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6809","nonce":2782033586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6811","nonce":2782033586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6815","nonce":2782033586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2782033586},{"type":"v1","addr":"172.21.15.113:6813","nonce":2782033586}]},"public_addr":"172.21.15.113:6809/2782033586","cluster_addr":"172.21.15.113:6811/2782033586","heartbeat_back_addr":"172.21.15.113:6815/2782033586","heartbeat_front_addr":"172.21.15.113:6813/2782033586","state":["exists","up"]},{"osd":6,"uuid":"a4e949a7-ba08-4503-adaa-6a987554323f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6817","nonce":3350371263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6819","nonce":3350371263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6823","nonce":3350371263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3350371263},{"type":"v1","addr":"172.21.15.113:6821","nonce":3350371263}]},"public_addr":"172.21.15.113:6817/3350371263","cluster_addr":"172.21.15.113:6819/3350371263","heartbeat_back_addr":"172.21.15.113:6823/3350371263","heartbeat_front_addr":"172.21.15.113:6821/3350371263","state":["exists","up"]},{"osd":7,"uuid":"1e5c240a-9909-4c1c-bdf4-a527f84170b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6825","nonce":2506384215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6827","nonce":2506384215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6831","nonce":2506384215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2506384215},{"type":"v1","addr":"172.21.15.113:6829","nonce":2506384215}]},"public_addr":"172.21.15.113:6825/2506384215","cluster_addr":"172.21.15.113:6827/2506384215","heartbeat_back_addr":"172.21.15.113:6831/2506384215","heartbeat_front_addr":"172.21.15.113:6829/2506384215","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:19:18.547544+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:19:45.487311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:20:13.883339+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:20:42.284248+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:21:04.054098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:21:28.385452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:21:52.670320+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:22:17.791344+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6801/293375230":"2024-04-23T22:18:15.224730+0000","172.21.15.47:6800/293375230":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/3586612862":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/869020271":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/2868740181":"2024-04-23T22:18:15.224730+0000","172.21.15.47:0/896685020":"2024-04-23T22:17:19.978457+0000","172.21.15.47:0/1128111817":"2024-04-23T22:17:19.978457+0000","172.21.15.47:6800/2320765223":"2024-04-23T22:17:19.978457+0000","172.21.15.47:0/1250921770":"2024-04-23T22:16:50.356523+0000","172.21.15.47:6801/2320765223":"2024-04-23T22:17:19.978457+0000","172.21.15.47:6800/680675122":"2024-04-23T22:16:50.356523+0000","172.21.15.47:0/1207367291":"2024-04-23T22:17:19.978457+0000","172.21.15.47:6801/680675122":"2024-04-23T22:16:50.356523+0000","172.21.15.47:0/3158220163":"2024-04-23T22:16:50.356523+0000","172.21.15.47:0/3814644933":"2024-04-23T22:16:50.356523+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T22:22:36.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:36 smithi047 ceph-mon[21846]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-04-22T22:22:36.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:36 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/540711530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:22:36.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:36 smithi113 ceph-mon[24083]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-04-22T22:22:36.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:36 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/540711530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:22:37.298 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T22:22:37.298 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T22:22:37.299 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T22:22:37.299 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T22:22:37.299 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T22:22:37.300 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T22:22:37.300 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-22T22:22:37.300 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-22T22:22:37.714 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:37.714 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:37.773 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:37.773 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:38.010 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:38.951 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:38 smithi047 ceph-mon[21846]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T22:22:38.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:38 smithi113 ceph-mon[24083]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T22:22:40.422 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:40.890 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:40 smithi047 ceph-mon[21846]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:40.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:40 smithi113 ceph-mon[24083]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:42.824 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:42 smithi047 ceph-mon[21846]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:42.956 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:42 smithi113 ceph-mon[24083]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:44.975 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:44 smithi047 ceph-mon[21846]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:44.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:44 smithi113 ceph-mon[24083]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:46.979 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:46 smithi047 ceph-mon[21846]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:46.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:46 smithi113 ceph-mon[24083]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:47.269 INFO:teuthology.orchestra.run.smithi047.stdout:115964117019 2024-04-22T22:22:47.269 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T22:22:47.923 INFO:teuthology.orchestra.run.smithi047.stdout:81604378656 2024-04-22T22:22:47.923 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T22:22:48.979 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:48 smithi047 ceph-mon[21846]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:48.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:48 smithi113 ceph-mon[24083]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:49.114 INFO:teuthology.orchestra.run.smithi047.stdout:240518168585 2024-04-22T22:22:49.114 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-22T22:22:50.582 INFO:teuthology.orchestra.run.smithi047.stdout:137438953495 2024-04-22T22:22:50.583 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T22:22:50.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:50 smithi047 ceph-mon[21846]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:50.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:50 smithi113 ceph-mon[24083]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:51.500 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:51.961 INFO:teuthology.orchestra.run.smithi047.stdout:171798691858 2024-04-22T22:22:51.961 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T22:22:52.932 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:52 smithi047 ceph-mon[21846]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:52.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:52 smithi113 ceph-mon[24083]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:53.676 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:53.676 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:53.677 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:22:54.391 INFO:teuthology.orchestra.run.smithi047.stdout:206158430223 2024-04-22T22:22:54.391 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-22T22:22:54.985 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:54 smithi047 ceph-mon[21846]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:54.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:54 smithi113 ceph-mon[24083]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:55.685 INFO:teuthology.orchestra.run.smithi047.stdout:60129542184 2024-04-22T22:22:55.686 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T22:22:56.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:56 smithi047 ceph-mon[21846]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:56.994 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:56 smithi113 ceph-mon[24083]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:57.622 INFO:teuthology.orchestra.run.smithi047.stdout:38654705711 2024-04-22T22:22:57.623 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T22:22:58.940 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:22:58 smithi047 ceph-mon[21846]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:22:58.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:22:58 smithi113 ceph-mon[24083]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:00.347 INFO:teuthology.orchestra.run.smithi047.stdout:115964117022 2024-04-22T22:23:00.611 INFO:teuthology.orchestra.run.smithi047.stdout:240518168587 2024-04-22T22:23:00.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:00 smithi047 ceph-mon[21846]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:00.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:00 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/4144737595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T22:23:00.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:00 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/3662262373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T22:23:00.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:00 smithi113 ceph-mon[24083]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:00.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:00 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/4144737595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T22:23:00.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:00 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/3662262373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T22:23:01.895 INFO:teuthology.orchestra.run.smithi047.stdout:81604378659 2024-04-22T22:23:02.238 INFO:teuthology.orchestra.run.smithi047.stdout:137438953497 2024-04-22T22:23:02.875 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:23:02.875 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:23:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:23:02.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:02 smithi047 ceph-mon[21846]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:02.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:02 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2848874368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T22:23:02.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:02 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2434393452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T22:23:02.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:02 smithi113 ceph-mon[24083]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:02.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:02 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2848874368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T22:23:02.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:02 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2434393452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T22:23:03.357 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117019 got 115964117022 for osd.3 2024-04-22T22:23:03.357 DEBUG:teuthology.parallel:result is None 2024-04-22T22:23:04.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:04 smithi047 ceph-mon[21846]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:04.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:04 smithi113 ceph-mon[24083]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:05.254 INFO:teuthology.orchestra.run.smithi047.stdout:171798691861 2024-04-22T22:23:05.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:05 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2094532224' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T22:23:05.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:05 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2094532224' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T22:23:05.997 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691858 got 171798691861 for osd.5 2024-04-22T22:23:05.998 DEBUG:teuthology.parallel:result is None 2024-04-22T22:23:06.541 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168585 got 240518168587 for osd.7 2024-04-22T22:23:06.541 DEBUG:teuthology.parallel:result is None 2024-04-22T22:23:06.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:06 smithi047 ceph-mon[21846]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:06.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:06 smithi113 ceph-mon[24083]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:08.252 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378656 got 81604378659 for osd.2 2024-04-22T22:23:08.253 DEBUG:teuthology.parallel:result is None 2024-04-22T22:23:08.619 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953495 got 137438953497 for osd.4 2024-04-22T22:23:08.619 DEBUG:teuthology.parallel:result is None 2024-04-22T22:23:08.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:08 smithi047 ceph-mon[21846]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:08.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:08 smithi113 ceph-mon[24083]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:09.911 INFO:teuthology.orchestra.run.smithi047.stdout:60129542187 2024-04-22T22:23:09.968 INFO:teuthology.orchestra.run.smithi047.stdout:206158430226 2024-04-22T22:23:10.709 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430223 got 206158430226 for osd.6 2024-04-22T22:23:10.710 DEBUG:teuthology.parallel:result is None 2024-04-22T22:23:10.959 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:10 smithi047 ceph-mon[21846]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:10.959 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:10 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/4031830103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T22:23:10.959 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:10 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/46181646' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T22:23:10.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:10 smithi113 ceph-mon[24083]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:10.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:10 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/4031830103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T22:23:10.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:10 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/46181646' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T22:23:11.364 INFO:teuthology.orchestra.run.smithi047.stdout:38654705713 2024-04-22T22:23:11.555 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542184 got 60129542187 for osd.1 2024-04-22T22:23:11.556 DEBUG:teuthology.parallel:result is None 2024-04-22T22:23:11.803 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:11 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/3289299413' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T22:23:12.231 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705711 got 38654705713 for osd.0 2024-04-22T22:23:12.231 DEBUG:teuthology.parallel:result is None 2024-04-22T22:23:12.231 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T22:23:12.232 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T22:23:12.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:11 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/3289299413' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T22:23:12.464 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:23:12.845 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:12 smithi047 ceph-mon[21846]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:13.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:12 smithi113 ceph-mon[24083]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:14.161 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:23:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-04-22T22:23:14.781 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-22T22:23:13.332051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48888,"kb_used_data":6392,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912328,"statfs":{"total":767960285184,"available":767910223872,"internally_reserved":0,"allocated":6545408,"data_stored":3572072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002146"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":222,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T22:22:23.699975+0000","last_change":"2024-04-22T22:22:23.699975+0000","last_active":"2024-04-22T22:22:23.699975+0000","last_peered":"2024-04-22T22:22:23.699975+0000","last_clean":"2024-04-22T22:22:23.699975+0000","last_became_active":"2024-04-22T22:22:23.699616+0000","last_became_peered":"2024-04-22T22:22:23.699616+0000","last_unstale":"2024-04-22T22:22:23.699975+0000","last_undegraded":"2024-04-22T22:22:23.699975+0000","last_fullsized":"2024-04-22T22:22:23.699975+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T22:20:17.836301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T22:20:17.836301+0000","last_clean_scrub_stamp":"2024-04-22T22:20:17.836301+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T07:54:00.375618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6060,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739092,"statfs":{"total":95995035648,"available":95988830208,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.498}]},{"osd":1,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":4.7460000000000004,"5min":4.7460000000000004,"15min":4.7460000000000004},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.375},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.438}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.34699999999999998,"15min":0.32100000000000001},"min":{"1min":0.247,"5min":0.185,"15min":0.185},"max":{"1min":0.501,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.35199999999999998,"15min":0.32200000000000001},"min":{"1min":0.221,"5min":0.192,"15min":0.192},"max":{"1min":0.60699999999999998,"5min":0.751,"15min":0.751},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Apr 22 22:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.40999999999999998,"15min":0.39600000000000002},"min":{"1min":0.35499999999999998,"5min":0.223,"15min":0.223},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"min":{"1min":0.36199999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.58599999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.26200000000000001}]},{"osd":3,"last update":"Mon Apr 22 22:22:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44500000000000001,"15min":0.44800000000000001},"min":{"1min":0.28699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.58199999999999996,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.499,"15min":0.501},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.35199999999999998}]},{"osd":4,"last update":"Mon Apr 22 22:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76700000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Apr 22 22:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon Apr 22 22:23:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 22:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.36799999999999999,"15min":0.34399999999999997},"min":{"1min":0.314,"5min":0.215,"15min":0.215},"max":{"1min":0.64100000000000001,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.33600000000000002},"min":{"1min":0.28899999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.57399999999999995,"5min":0.877,"15min":0.877},"last":0.36599999999999999}]},{"osd":2,"last update":"Mon Apr 22 22:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.33200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.63600000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.377,"5min":0.245,"15min":0.245},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.51600000000000001}]},{"osd":3,"last update":"Mon Apr 22 22:22:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.434},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.34399999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.42199999999999999}]},{"osd":4,"last update":"Mon Apr 22 22:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60999999999999999,"15min":0.60299999999999998},"min":{"1min":0.46999999999999997,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.627},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.58899999999999997,"15min":0.58399999999999996},"min":{"1min":0.47299999999999998,"5min":0.442,"15min":0.442},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Apr 22 22:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.505},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57799999999999996}]},{"osd":6,"last update":"Mon Apr 22 22:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6244,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738908,"statfs":{"total":95995035648,"available":95988641792,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.27000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.61799999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Apr 22 22:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.39700000000000002,"15min":0.38500000000000001},"min":{"1min":0.32300000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.39000000000000001,"15min":0.38300000000000001},"min":{"1min":0.308,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.34300000000000003}]},{"osd":3,"last update":"Mon Apr 22 22:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.432},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.36199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.63800000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.39100000000000001}]},{"osd":4,"last update":"Mon Apr 22 22:23:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.55900000000000005,"15min":0.54500000000000004},"min":{"1min":0.42299999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68300000000000005}]},{"osd":5,"last update":"Mon Apr 22 22:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.76400000000000001}]},{"osd":6,"last update":"Mon Apr 22 22:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.76900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]}]},{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38900000000000001,"15min":0.39300000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.48899999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46200000000000002,"15min":0.47099999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.498}]},{"osd":1,"last update":"Mon Apr 22 22:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.61399999999999999,"5min":0.751,"15min":0.751},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45800000000000002,"15min":0.45900000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.61599999999999999,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.53400000000000003}]},{"osd":2,"last update":"Mon Apr 22 22:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47199999999999998,"15min":0.47499999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48399999999999999,"15min":0.48799999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.624,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.52000000000000002}]},{"osd":4,"last update":"Mon Apr 22 22:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.67000000000000004}]},{"osd":5,"last update":"Mon Apr 22 22:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.77600000000000002}]},{"osd":6,"last update":"Mon Apr 22 22:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953500,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59699999999999998,"15min":0.60499999999999998},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70199999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.438},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72899999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Apr 22 22:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57799999999999996,"15min":0.58099999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80900000000000005,"5min":0.879,"15min":0.879},"last":0.41899999999999998}]},{"osd":2,"last update":"Mon Apr 22 22:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61399999999999999,"15min":0.621},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.74199999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.52100000000000002}]},{"osd":3,"last update":"Mon Apr 22 22:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.80200000000000005,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.63600000000000001,"15min":0.64100000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.79000000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.499}]},{"osd":5,"last update":"Mon Apr 22 22:22:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Apr 22 22:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.47199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]}]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5668,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739484,"statfs":{"total":95995035648,"available":95989231616,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:22:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.58999999999999997}]},{"osd":1,"last update":"Mon Apr 22 22:22:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Apr 22 22:22:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Mon Apr 22 22:22:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.46700000000000003}]},{"osd":4,"last update":"Mon Apr 22 22:22:32 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Apr 22 22:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.63200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T22:23:14.782 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T22:23:14.891 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:14 smithi047 ceph-mon[21846]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:15.003 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:23:15.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:14 smithi113 ceph-mon[24083]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:16.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:15 smithi047 ceph-mon[21846]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:16.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:23:16.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:23:16.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:15 smithi113 ceph-mon[24083]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:16.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:23:16.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:23:16.809 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:23:16.811 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-04-22T22:23:17.127 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:16 smithi047 ceph-mon[21846]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:17.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:16 smithi113 ceph-mon[24083]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:17.460 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-04-22T22:23:15.332475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48888,"kb_used_data":6392,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912328,"statfs":{"total":767960285184,"available":767910223872,"internally_reserved":0,"allocated":6545408,"data_stored":3572072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002030"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":222,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T22:22:23.699975+0000","last_change":"2024-04-22T22:22:23.699975+0000","last_active":"2024-04-22T22:22:23.699975+0000","last_peered":"2024-04-22T22:22:23.699975+0000","last_clean":"2024-04-22T22:22:23.699975+0000","last_became_active":"2024-04-22T22:22:23.699616+0000","last_became_peered":"2024-04-22T22:22:23.699616+0000","last_unstale":"2024-04-22T22:22:23.699975+0000","last_undegraded":"2024-04-22T22:22:23.699975+0000","last_fullsized":"2024-04-22T22:22:23.699975+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T22:20:17.836301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T22:20:17.836301+0000","last_clean_scrub_stamp":"2024-04-22T22:20:17.836301+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T07:54:00.375618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6060,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739092,"statfs":{"total":95995035648,"available":95988830208,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.498}]},{"osd":1,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":4.7460000000000004,"5min":4.7460000000000004,"15min":4.7460000000000004},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.375},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.438}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.34699999999999998,"15min":0.32100000000000001},"min":{"1min":0.247,"5min":0.185,"15min":0.185},"max":{"1min":0.501,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.35199999999999998,"15min":0.32200000000000001},"min":{"1min":0.221,"5min":0.192,"15min":0.192},"max":{"1min":0.60699999999999998,"5min":0.751,"15min":0.751},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Apr 22 22:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.40999999999999998,"15min":0.39600000000000002},"min":{"1min":0.35499999999999998,"5min":0.223,"15min":0.223},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"min":{"1min":0.36199999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.58599999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.26200000000000001}]},{"osd":3,"last update":"Mon Apr 22 22:22:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44500000000000001,"15min":0.44800000000000001},"min":{"1min":0.28699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.58199999999999996,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.499,"15min":0.501},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.35199999999999998}]},{"osd":4,"last update":"Mon Apr 22 22:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76700000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Apr 22 22:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon Apr 22 22:23:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 22:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.36799999999999999,"15min":0.34399999999999997},"min":{"1min":0.314,"5min":0.215,"15min":0.215},"max":{"1min":0.64100000000000001,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.33600000000000002},"min":{"1min":0.28899999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.57399999999999995,"5min":0.877,"15min":0.877},"last":0.36599999999999999}]},{"osd":2,"last update":"Mon Apr 22 22:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.33200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.63600000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.377,"5min":0.245,"15min":0.245},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.51600000000000001}]},{"osd":3,"last update":"Mon Apr 22 22:22:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.434},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.34399999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.42199999999999999}]},{"osd":4,"last update":"Mon Apr 22 22:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60999999999999999,"15min":0.60299999999999998},"min":{"1min":0.46999999999999997,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.627},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.58899999999999997,"15min":0.58399999999999996},"min":{"1min":0.47299999999999998,"5min":0.442,"15min":0.442},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Apr 22 22:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.505},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57799999999999996}]},{"osd":6,"last update":"Mon Apr 22 22:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6244,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738908,"statfs":{"total":95995035648,"available":95988641792,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.27000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.61799999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Apr 22 22:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.39700000000000002,"15min":0.38500000000000001},"min":{"1min":0.32300000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.39000000000000001,"15min":0.38300000000000001},"min":{"1min":0.308,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.34300000000000003}]},{"osd":3,"last update":"Mon Apr 22 22:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.432},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.36199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.63800000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.39100000000000001}]},{"osd":4,"last update":"Mon Apr 22 22:23:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.55900000000000005,"15min":0.54500000000000004},"min":{"1min":0.42299999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68300000000000005}]},{"osd":5,"last update":"Mon Apr 22 22:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.76400000000000001}]},{"osd":6,"last update":"Mon Apr 22 22:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.76900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]}]},{"osd":3,"up_from":27,"seq":115964117026,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38900000000000001,"15min":0.39300000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.48899999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46200000000000002,"15min":0.47099999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.40500000000000003}]},{"osd":1,"last update":"Mon Apr 22 22:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.61399999999999999,"5min":0.751,"15min":0.751},"last":0.308},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45800000000000002,"15min":0.45900000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.61599999999999999,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon Apr 22 22:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47199999999999998,"15min":0.47499999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48399999999999999,"15min":0.48799999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.624,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Apr 22 22:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.59999999999999998,"15min":0.59399999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Apr 22 22:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.59199999999999997}]},{"osd":6,"last update":"Mon Apr 22 22:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953500,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59699999999999998,"15min":0.60499999999999998},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70199999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.438},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72899999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Apr 22 22:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57799999999999996,"15min":0.58099999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80900000000000005,"5min":0.879,"15min":0.879},"last":0.41899999999999998}]},{"osd":2,"last update":"Mon Apr 22 22:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61399999999999999,"15min":0.621},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.74199999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.52100000000000002}]},{"osd":3,"last update":"Mon Apr 22 22:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.80200000000000005,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.63600000000000001,"15min":0.64100000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.79000000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.499}]},{"osd":5,"last update":"Mon Apr 22 22:22:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Apr 22 22:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.47199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]}]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5668,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739484,"statfs":{"total":95995035648,"available":95989231616,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:22:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.58999999999999997}]},{"osd":1,"last update":"Mon Apr 22 22:22:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Apr 22 22:22:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Mon Apr 22 22:22:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.46700000000000003}]},{"osd":4,"last update":"Mon Apr 22 22:22:32 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Apr 22 22:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.63200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T22:23:17.462 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T22:23:17.462 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T22:23:17.462 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T22:23:17.462 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T22:23:17.684 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/mon.smithi047/config 2024-04-22T22:23:18.105 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:17 smithi047 ceph-mon[21846]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:18.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:17 smithi113 ceph-mon[24083]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:18.986 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:18 smithi047 ceph-mon[21846]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:19.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:18 smithi113 ceph-mon[24083]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:19.410 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:23:19.410 INFO:teuthology.orchestra.run.smithi047.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T22:23:20.017 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:19 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/4294140063' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T22:23:20.040 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T22:23:20.041 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T22:23:20.041 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T22:23:20.051 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi047.front.sepia.ceph.com 2024-04-22T22:23:20.051 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T22:23:20.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:19 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/4294140063' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T22:23:21.021 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:20 smithi047 ceph-mon[21846]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:21.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:20 smithi113 ceph-mon[24083]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:21.836 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:23:21.836 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:21.836 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:21 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:23:21.874 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:23:21.874 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:21.874 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:21 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:23:22.019 INFO:teuthology.orchestra.run.smithi047.stdout:Backend: cephadm 2024-04-22T22:23:22.019 INFO:teuthology.orchestra.run.smithi047.stdout:Available: Yes 2024-04-22T22:23:22.019 INFO:teuthology.orchestra.run.smithi047.stdout:Paused: No 2024-04-22T22:23:22.594 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T22:23:23.098 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:22 smithi047 ceph-mon[21846]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:23.098 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:22 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:23.098 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:22 smithi047 ceph-mon[21846]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:23.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:22 smithi113 ceph-mon[24083]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:23.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:22 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:23.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:22 smithi113 ceph-mon[24083]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:24.487 INFO:teuthology.orchestra.run.smithi047.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T22:23:24.487 INFO:teuthology.orchestra.run.smithi047.stdout:alertmanager.smithi047 smithi047 *:9093,9094 running (4m) 2m ago 5m 15.5M - ba2b418f427c 11fa9d9fcc72 2024-04-22T22:23:24.487 INFO:teuthology.orchestra.run.smithi047.stdout:crash.smithi047 smithi047 running (5m) 2m ago 5m 6409k - 17.2.0 e1d6a67b021e 1b99a2e1b2ad 2024-04-22T22:23:24.487 INFO:teuthology.orchestra.run.smithi047.stdout:crash.smithi113 smithi113 running (5m) 65s ago 5m 6400k - 17.2.0 e1d6a67b021e a35f59de8f2d 2024-04-22T22:23:24.487 INFO:teuthology.orchestra.run.smithi047.stdout:grafana.smithi047 smithi047 *:3000 running (3m) 2m ago 5m 46.0M - 8.3.5 dad864ee21e9 968e836a6db0 2024-04-22T22:23:24.487 INFO:teuthology.orchestra.run.smithi047.stdout:mgr.smithi047.uutpnj smithi047 *:9283 running (6m) 2m ago 6m 448M - 17.2.0 e1d6a67b021e 42f8f60aedc0 2024-04-22T22:23:24.487 INFO:teuthology.orchestra.run.smithi047.stdout:mgr.smithi113.qvxssq smithi113 *:8443,9283 running (4m) 65s ago 4m 413M - 17.2.0 e1d6a67b021e 7cab77c0494d 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:mon.smithi047 smithi047 running (6m) 2m ago 7m 41.7M 2048M 17.2.0 e1d6a67b021e 96cda5acb635 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:mon.smithi113 smithi113 running (4m) 65s ago 4m 39.7M 2048M 17.2.0 e1d6a67b021e 82b94d1b7aaf 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:node-exporter.smithi047 smithi047 *:9100 running (5m) 2m ago 5m 21.1M - 1dbe0e931976 618a9c0cf648 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:node-exporter.smithi113 smithi113 *:9100 running (4m) 65s ago 4m 16.8M - 1dbe0e931976 495bdeae71f6 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.0 smithi047 running (4m) 2m ago 4m 34.1M 3217M 17.2.0 e1d6a67b021e 12042fdd6c37 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.1 smithi047 running (3m) 2m ago 3m 38.1M 3217M 17.2.0 e1d6a67b021e 077c4bd9730e 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.2 smithi047 running (3m) 2m ago 3m 32.1M 3217M 17.2.0 e1d6a67b021e 5a898b3c4e7b 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.3 smithi047 running (2m) 2m ago 2m 25.4M 3217M 17.2.0 e1d6a67b021e 31f106431cce 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.4 smithi113 running (2m) 65s ago 2m 34.3M 3984M 17.2.0 e1d6a67b021e 6f83763ef7dc 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.5 smithi113 running (2m) 65s ago 2m 35.6M 3984M 17.2.0 e1d6a67b021e e7cc308c1735 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.6 smithi113 running (96s) 65s ago 95s 33.7M 3984M 17.2.0 e1d6a67b021e 87b85a4c560e 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.7 smithi113 running (71s) 65s ago 70s 10.4M 3984M 17.2.0 e1d6a67b021e 8ab7d8d39212 2024-04-22T22:23:24.488 INFO:teuthology.orchestra.run.smithi047.stdout:prometheus.smithi047 smithi047 *:9095 running (4m) 2m ago 4m 53.4M - 514e6a882f6e d2f72369ca5a 2024-04-22T22:23:25.124 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:24 smithi047 ceph-mon[21846]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:25.153 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T22:23:25.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:24 smithi113 ceph-mon[24083]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:26.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:25 smithi047 ceph-mon[21846]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:26.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:25 smithi113 ceph-mon[24083]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:27.114 INFO:teuthology.orchestra.run.smithi047.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T22:23:27.114 INFO:teuthology.orchestra.run.smithi047.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-22T22:23:27.114 INFO:teuthology.orchestra.run.smithi047.stdout:crash 2/2 2m ago 6m * 2024-04-22T22:23:27.114 INFO:teuthology.orchestra.run.smithi047.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-22T22:23:27.114 INFO:teuthology.orchestra.run.smithi047.stdout:mgr 2/2 2m ago 6m count:2 2024-04-22T22:23:27.114 INFO:teuthology.orchestra.run.smithi047.stdout:mon 2/2 2m ago 5m smithi047:172.21.15.47=smithi047;smithi113:172.21.15.113=smithi113;count:2 2024-04-22T22:23:27.114 INFO:teuthology.orchestra.run.smithi047.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-22T22:23:27.114 INFO:teuthology.orchestra.run.smithi047.stdout:osd 8 2m ago - 2024-04-22T22:23:27.114 INFO:teuthology.orchestra.run.smithi047.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-22T22:23:27.115 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:26 smithi047 ceph-mon[21846]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:27.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:26 smithi113 ceph-mon[24083]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:27.703 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T22:23:28.953 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:28 smithi047 ceph-mon[21846]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:28.953 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:28 smithi047 ceph-mon[21846]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:29.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:28 smithi113 ceph-mon[24083]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:29.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:28 smithi113 ceph-mon[24083]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:29.600 INFO:teuthology.orchestra.run.smithi047.stdout:HOST ADDR LABELS STATUS 2024-04-22T22:23:29.600 INFO:teuthology.orchestra.run.smithi047.stdout:smithi047 172.21.15.47 2024-04-22T22:23:29.600 INFO:teuthology.orchestra.run.smithi047.stdout:smithi113 172.21.15.113 2024-04-22T22:23:29.600 INFO:teuthology.orchestra.run.smithi047.stdout:2 hosts in cluster 2024-04-22T22:23:30.278 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T22:23:31.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:30 smithi113 ceph-mon[24083]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:31.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:30 smithi113 ceph-mon[24083]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:31.295 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:30 smithi047 ceph-mon[21846]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:31.295 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:30 smithi047 ceph-mon[21846]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:32.404 INFO:teuthology.orchestra.run.smithi047.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-04-22T22:23:32.404 INFO:teuthology.orchestra.run.smithi047.stdout:smithi047 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75130022480DGN 480G LVM detected, locked 2024-04-22T22:23:32.404 INFO:teuthology.orchestra.run.smithi047.stdout:smithi113 /dev/nvme0n1 ssd INTEL_SSDPEDMX020T4_CVPD5171000M2P0SGN 2000G LVM detected, locked 2024-04-22T22:23:32.907 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-22T22:23:32.919 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi047.front.sepia.ceph.com 2024-04-22T22:23:32.920 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-22T22:23:32.944 INFO:teuthology.orchestra.run.smithi047.stderr:+ systemctl stop nfs-server 2024-04-22T22:23:32.951 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:32 smithi047 ceph-mon[21846]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:32.952 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi113.front.sepia.ceph.com 2024-04-22T22:23:32.952 DEBUG:teuthology.orchestra.run.smithi113:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-22T22:23:32.980 INFO:teuthology.orchestra.run.smithi113.stderr:+ systemctl stop nfs-server 2024-04-22T22:23:32.988 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T22:23:32.996 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi047.front.sepia.ceph.com 2024-04-22T22:23:32.997 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- bash -c 'ceph fs volume create foofs' 2024-04-22T22:23:33.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:32 smithi113 ceph-mon[24083]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:34.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:33 smithi047 ceph-mon[21846]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:34.245 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:33 smithi113 ceph-mon[24083]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:35.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:34 smithi113 ceph-mon[24083]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:35.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:34 smithi047 ceph-mon[21846]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:36.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:35 smithi113 ceph-mon[24083]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:36.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:35 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-22T22:23:36.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:35 smithi047 ceph-mon[21846]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:36.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:35 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-22T22:23:37.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:36 smithi113 ceph-mon[24083]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:37.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:36 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-22T22:23:37.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:36 smithi113 ceph-mon[24083]: osdmap e61: 8 total, 8 up, 8 in 2024-04-22T22:23:37.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:36 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-22T22:23:37.277 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:36 smithi047 ceph-mon[21846]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:37.277 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:36 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-22T22:23:37.277 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:36 smithi047 ceph-mon[21846]: osdmap e61: 8 total, 8 up, 8 in 2024-04-22T22:23:37.278 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:36 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-22T22:23:37.278 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:36 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047[21841]: 2024-04-22T22:23:36.942+0000 7f3a7d299700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T22:23:37.619 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-04-22T22:23:37.629 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-04-22T22:23:37.629 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-22T22:23:38.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-22T22:23:38.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: osdmap e62: 8 total, 8 up, 8 in 2024-04-22T22:23:38.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-22T22:23:38.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T22:23:38.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T22:23:38.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: osdmap e63: 8 total, 8 up, 8 in 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: fsmap foofs:0 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: Saving service mds.foofs spec with placement count:2 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi047.avrnmr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi047.avrnmr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:38.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:37 smithi047 ceph-mon[21846]: Deploying daemon mds.foofs.smithi047.avrnmr on smithi047 2024-04-22T22:23:38.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-22T22:23:38.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: osdmap e62: 8 total, 8 up, 8 in 2024-04-22T22:23:38.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-22T22:23:38.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T22:23:38.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T22:23:38.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: osdmap e63: 8 total, 8 up, 8 in 2024-04-22T22:23:38.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-22T22:23:38.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: fsmap foofs:0 2024-04-22T22:23:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: Saving service mds.foofs spec with placement count:2 2024-04-22T22:23:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:23:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:23:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi047.avrnmr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T22:23:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi047.avrnmr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T22:23:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:37 smithi113 ceph-mon[24083]: Deploying daemon mds.foofs.smithi047.avrnmr on smithi047 2024-04-22T22:23:39.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:38 smithi047 ceph-mon[21846]: pgmap v212: 65 pgs: 8 creating+peering, 2 active+clean, 55 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:39.052 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:38 smithi047 ceph-mon[21846]: osdmap e64: 8 total, 8 up, 8 in 2024-04-22T22:23:39.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:38 smithi113 ceph-mon[24083]: pgmap v212: 65 pgs: 8 creating+peering, 2 active+clean, 55 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:39.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:38 smithi113 ceph-mon[24083]: osdmap e64: 8 total, 8 up, 8 in 2024-04-22T22:23:40.242 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:39 smithi047 ceph-mon[21846]: osdmap e65: 8 total, 8 up, 8 in 2024-04-22T22:23:40.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:39 smithi113 ceph-mon[24083]: osdmap e65: 8 total, 8 up, 8 in 2024-04-22T22:23:40.637 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:23:40.637 INFO:teuthology.orchestra.run.smithi047.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-22T22:17:09.145467Z", "last_refresh": "2024-04-22T22:20:52.564617Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-22T22:17:02.897190Z", "last_refresh": "2024-04-22T22:20:52.564686Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-22T22:17:05.743394Z", "last_refresh": "2024-04-22T22:20:52.564753Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-22T22:23:36.961509Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-22T22:23:36.957046Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-22T22:17:00.976780Z", "last_refresh": "2024-04-22T22:20:52.564542Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi047:172.21.15.47=smithi047", "smithi113:172.21.15.113=smithi113"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-22T22:17:51.581181Z", "last_refresh": "2024-04-22T22:20:52.564397Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-22T22:17:07.036780Z", "last_refresh": "2024-04-22T22:20:52.564817Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-22T22:20:52.564960Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-22T22:17:04.096157Z", "last_refresh": "2024-04-22T22:20:52.564895Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-22T22:23:41.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:40 smithi113 ceph-mon[24083]: pgmap v215: 65 pgs: 8 creating+peering, 11 active+clean, 46 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:41.250 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:40 smithi047 ceph-mon[21846]: pgmap v215: 65 pgs: 8 creating+peering, 11 active+clean, 46 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:41.496 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-22T22:23:42.141 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:41 smithi113 ceph-mon[24083]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:42.141 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:41 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:42.141 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:41 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi113.rtxmqv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T22:23:42.141 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:41 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi113.rtxmqv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T22:23:42.141 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:41 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:42.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:41 smithi047 ceph-mon[21846]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:42.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:41 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:42.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:41 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi113.rtxmqv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T22:23:42.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:41 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi113.rtxmqv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T22:23:42.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:41 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:42.497 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-22T22:23:43.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:42 smithi047 ceph-mon[21846]: pgmap v216: 65 pgs: 8 creating+peering, 57 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:43.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:42 smithi047 ceph-mon[21846]: Deploying daemon mds.foofs.smithi113.rtxmqv on smithi113 2024-04-22T22:23:43.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:42 smithi047 ceph-mon[21846]: mds.? [v2:172.21.15.47:6834/4140202189,v1:172.21.15.47:6835/4140202189] up:boot 2024-04-22T22:23:43.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:42 smithi047 ceph-mon[21846]: daemon mds.foofs.smithi047.avrnmr assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-22T22:23:43.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:42 smithi047 ceph-mon[21846]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T22:23:43.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:42 smithi047 ceph-mon[21846]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T22:23:43.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:42 smithi047 ceph-mon[21846]: Cluster is now healthy 2024-04-22T22:23:43.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:42 smithi047 ceph-mon[21846]: fsmap foofs:0 1 up:standby 2024-04-22T22:23:43.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:42 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi047.avrnmr"}]: dispatch 2024-04-22T22:23:43.073 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:42 smithi047 ceph-mon[21846]: fsmap foofs:1 {0=foofs.smithi047.avrnmr=up:creating} 2024-04-22T22:23:43.074 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:42 smithi047 ceph-mon[21846]: daemon mds.foofs.smithi047.avrnmr is now active in filesystem foofs as rank 0 2024-04-22T22:23:43.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:42 smithi113 ceph-mon[24083]: pgmap v216: 65 pgs: 8 creating+peering, 57 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:23:43.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:42 smithi113 ceph-mon[24083]: Deploying daemon mds.foofs.smithi113.rtxmqv on smithi113 2024-04-22T22:23:43.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:42 smithi113 ceph-mon[24083]: mds.? [v2:172.21.15.47:6834/4140202189,v1:172.21.15.47:6835/4140202189] up:boot 2024-04-22T22:23:43.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:42 smithi113 ceph-mon[24083]: daemon mds.foofs.smithi047.avrnmr assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-22T22:23:43.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:42 smithi113 ceph-mon[24083]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T22:23:43.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:42 smithi113 ceph-mon[24083]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T22:23:43.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:42 smithi113 ceph-mon[24083]: Cluster is now healthy 2024-04-22T22:23:43.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:42 smithi113 ceph-mon[24083]: fsmap foofs:0 1 up:standby 2024-04-22T22:23:43.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:42 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi047.avrnmr"}]: dispatch 2024-04-22T22:23:43.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:42 smithi113 ceph-mon[24083]: fsmap foofs:1 {0=foofs.smithi047.avrnmr=up:creating} 2024-04-22T22:23:43.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:42 smithi113 ceph-mon[24083]: daemon mds.foofs.smithi047.avrnmr is now active in filesystem foofs as rank 0 2024-04-22T22:23:44.014 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:43 smithi113 ceph-mon[24083]: mds.? [v2:172.21.15.47:6834/4140202189,v1:172.21.15.47:6835/4140202189] up:active 2024-04-22T22:23:44.014 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:43 smithi113 ceph-mon[24083]: fsmap foofs:1 {0=foofs.smithi047.avrnmr=up:active} 2024-04-22T22:23:44.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:43 smithi047 ceph-mon[21846]: mds.? [v2:172.21.15.47:6834/4140202189,v1:172.21.15.47:6835/4140202189] up:active 2024-04-22T22:23:44.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:43 smithi047 ceph-mon[21846]: fsmap foofs:1 {0=foofs.smithi047.avrnmr=up:active} 2024-04-22T22:23:44.674 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:23:44.674 INFO:teuthology.orchestra.run.smithi047.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-22T22:17:09.145467Z", "last_refresh": "2024-04-22T22:20:52.564617Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-22T22:17:02.897190Z", "last_refresh": "2024-04-22T22:20:52.564686Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-22T22:17:05.743394Z", "last_refresh": "2024-04-22T22:20:52.564753Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-22T22:23:36.961509Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-22T22:23:36.957046Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-22T22:17:00.976780Z", "last_refresh": "2024-04-22T22:20:52.564542Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi047:172.21.15.47=smithi047", "smithi113:172.21.15.113=smithi113"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-22T22:17:51.581181Z", "last_refresh": "2024-04-22T22:20:52.564397Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-22T22:17:07.036780Z", "last_refresh": "2024-04-22T22:20:52.564817Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-22T22:20:52.564960Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-22T22:17:04.096157Z", "last_refresh": "2024-04-22T22:20:52.564895Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-22T22:23:45.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:45 smithi047 ceph-mon[21846]: pgmap v217: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 480 B/s wr, 0 op/s 2024-04-22T22:23:45.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:45.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:23:45.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:45.392 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:23:45.393 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:45 smithi047 ceph-mon[21846]: mds.? [v2:172.21.15.113:6832/3991300797,v1:172.21.15.113:6833/3991300797] up:boot 2024-04-22T22:23:45.393 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:45 smithi047 ceph-mon[21846]: fsmap foofs:1 {0=foofs.smithi047.avrnmr=up:active} 1 up:standby 2024-04-22T22:23:45.393 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:45 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi113.rtxmqv"}]: dispatch 2024-04-22T22:23:45.393 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:45 smithi047 ceph-mon[21846]: fsmap foofs:1 {0=foofs.smithi047.avrnmr=up:active} 1 up:standby 2024-04-22T22:23:45.428 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:45 smithi113 ceph-mon[24083]: pgmap v217: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 480 B/s wr, 0 op/s 2024-04-22T22:23:45.428 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:45.428 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:23:45.428 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:45.428 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:23:45.428 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:45 smithi113 ceph-mon[24083]: mds.? [v2:172.21.15.113:6832/3991300797,v1:172.21.15.113:6833/3991300797] up:boot 2024-04-22T22:23:45.429 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:45 smithi113 ceph-mon[24083]: fsmap foofs:1 {0=foofs.smithi047.avrnmr=up:active} 1 up:standby 2024-04-22T22:23:45.429 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:45 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi113.rtxmqv"}]: dispatch 2024-04-22T22:23:45.429 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:45 smithi113 ceph-mon[24083]: fsmap foofs:1 {0=foofs.smithi047.avrnmr=up:active} 1 up:standby 2024-04-22T22:23:45.611 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-22T22:23:46.383 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:46 smithi047 ceph-mon[21846]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:46.383 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:46 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:46.396 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:46 smithi113 ceph-mon[24083]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:46.396 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:46 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:46.611 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-22T22:23:47.295 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:47 smithi047 ceph-mon[21846]: pgmap v218: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 383 B/s wr, 0 op/s 2024-04-22T22:23:47.295 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:47 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:47.496 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:47 smithi113 ceph-mon[24083]: pgmap v218: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 383 B/s wr, 0 op/s 2024-04-22T22:23:47.496 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:47 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:48.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:48 smithi047 ceph-mon[21846]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-22T22:23:48.496 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:48 smithi113 ceph-mon[24083]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-22T22:23:49.741 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:23:49.742 INFO:teuthology.orchestra.run.smithi047.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-22T22:17:09.145467Z", "last_refresh": "2024-04-22T22:20:52.564617Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-22T22:17:02.897190Z", "last_refresh": "2024-04-22T22:20:52.564686Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-22T22:17:05.743394Z", "last_refresh": "2024-04-22T22:20:52.564753Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-22T22:23:36.961509Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-22T22:23:36.957046Z", "last_refresh": "2024-04-22T22:23:46.127465Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-22T22:17:00.976780Z", "last_refresh": "2024-04-22T22:20:52.564542Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi047:172.21.15.47=smithi047", "smithi113:172.21.15.113=smithi113"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-22T22:17:51.581181Z", "last_refresh": "2024-04-22T22:20:52.564397Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-22T22:17:07.036780Z", "last_refresh": "2024-04-22T22:20:52.564817Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-22T22:20:52.564960Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-22T22:17:04.096157Z", "last_refresh": "2024-04-22T22:20:52.564895Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-22T22:23:50.313 INFO:tasks.cephadm:mds.foofs has 1/2 2024-04-22T22:23:50.565 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:50 smithi047 ceph-mon[21846]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-22T22:23:50.565 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:50 smithi047 ceph-mon[21846]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:50.565 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:50 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:50.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:50 smithi113 ceph-mon[24083]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-22T22:23:50.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:50 smithi113 ceph-mon[24083]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:50.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:50 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:51.315 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-22T22:23:52.643 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:52 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:52.643 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:52 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:52.643 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:52 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:23:52.643 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:52 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:52.643 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:52 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:23:52.643 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:52 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:52.643 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:52 smithi047 ceph-mon[21846]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-22T22:23:52.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:52 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:52.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:52 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:52.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:52 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:23:52.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:52 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:52.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:52 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:23:52.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:52 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:52.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:52 smithi113 ceph-mon[24083]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-22T22:23:53.428 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:23:53.428 INFO:teuthology.orchestra.run.smithi047.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-22T22:17:09.145467Z", "last_refresh": "2024-04-22T22:23:51.275741Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-22T22:17:02.897190Z", "last_refresh": "2024-04-22T22:23:46.126871Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-22T22:17:05.743394Z", "last_refresh": "2024-04-22T22:23:51.275892Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-22T22:23:36.961509Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-22T22:23:36.957046Z", "last_refresh": "2024-04-22T22:23:46.127465Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-22T22:17:00.976780Z", "last_refresh": "2024-04-22T22:23:46.127007Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi047:172.21.15.47=smithi047", "smithi113:172.21.15.113=smithi113"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-22T22:17:51.581181Z", "last_refresh": "2024-04-22T22:23:46.127080Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-22T22:17:07.036780Z", "last_refresh": "2024-04-22T22:23:46.127146Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-22T22:23:46.127211Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-22T22:17:04.096157Z", "last_refresh": "2024-04-22T22:23:51.276020Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-22T22:23:54.262 INFO:tasks.cephadm:mds.foofs has 2/2 2024-04-22T22:23:54.263 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T22:23:54.272 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi047.front.sepia.ceph.com 2024-04-22T22:23:54.272 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-04-22T22:23:54.586 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:54 smithi047 ceph-mon[21846]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-22T22:23:54.586 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:54 smithi047 ceph-mon[21846]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:54.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:54 smithi113 ceph-mon[24083]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-22T22:23:54.747 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:54 smithi113 ceph-mon[24083]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:23:56.746 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:56 smithi113 ceph-mon[24083]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 853 B/s wr, 3 op/s 2024-04-22T22:23:56.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:56 smithi047 ceph-mon[21846]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 853 B/s wr, 3 op/s 2024-04-22T22:23:57.710 INFO:teuthology.orchestra.run.smithi047.stdout:NFS Cluster Created Successfully 2024-04-22T22:23:57.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:57 smithi047 ceph-mon[21846]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:57.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:57 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-22T22:23:57.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:57 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-22T22:23:57.879 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:57 smithi047 ceph-mon[21846]: osdmap e66: 8 total, 8 up, 8 in 2024-04-22T22:23:57.880 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:57 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-22T22:23:57.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:57 smithi113 ceph-mon[24083]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:23:57.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:57 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-22T22:23:57.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:57 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-22T22:23:57.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:57 smithi113 ceph-mon[24083]: osdmap e66: 8 total, 8 up, 8 in 2024-04-22T22:23:57.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:57 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-22T22:23:58.310 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-04-22T22:23:58.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: pgmap v225: 97 pgs: 9 creating+peering, 23 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-22T22:23:58.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-22T22:23:58.894 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: osdmap e67: 8 total, 8 up, 8 in 2024-04-22T22:23:58.894 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: Saving service nfs.foo spec with placement count:2 2024-04-22T22:23:58.894 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:58.894 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:23:58.894 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:58.894 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:23:58.894 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:58.895 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:58.895 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: Creating key for client.nfs.foo.0.0.smithi047.vfwjww 2024-04-22T22:23:58.895 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi047.vfwjww", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-22T22:23:58.895 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi047.vfwjww", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-22T22:23:58.895 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-22T22:23:58.895 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-22T22:23:58.896 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-22T22:23:58.896 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:58.896 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-22T22:23:58.896 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-22T22:23:58.896 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: Rados config object exists: conf-nfs.foo 2024-04-22T22:23:58.896 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: Creating key for client.nfs.foo.0.0.smithi047.vfwjww-rgw 2024-04-22T22:23:58.896 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi047.vfwjww-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:23:58.897 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi047.vfwjww-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:23:58.897 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:58.897 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:58 smithi047 ceph-mon[21846]: Deploying daemon nfs.foo.0.0.smithi047.vfwjww on smithi047 2024-04-22T22:23:58.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: pgmap v225: 97 pgs: 9 creating+peering, 23 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: osdmap e67: 8 total, 8 up, 8 in 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: Saving service nfs.foo spec with placement count:2 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: Creating key for client.nfs.foo.0.0.smithi047.vfwjww 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi047.vfwjww", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi047.vfwjww", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-22T22:23:58.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-22T22:23:58.998 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-22T22:23:58.998 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:58.998 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-22T22:23:58.998 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-22T22:23:58.998 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: Rados config object exists: conf-nfs.foo 2024-04-22T22:23:58.998 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: Creating key for client.nfs.foo.0.0.smithi047.vfwjww-rgw 2024-04-22T22:23:58.998 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi047.vfwjww-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:23:58.998 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi047.vfwjww-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:23:58.998 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:23:58.998 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:58 smithi113 ceph-mon[24083]: Deploying daemon nfs.foo.0.0.smithi047.vfwjww on smithi047 2024-04-22T22:23:59.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:23:59 smithi113 ceph-mon[24083]: osdmap e68: 8 total, 8 up, 8 in 2024-04-22T22:24:00.137 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:23:59 smithi047 ceph-mon[21846]: osdmap e68: 8 total, 8 up, 8 in 2024-04-22T22:24:00.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:00 smithi047 ceph-mon[21846]: pgmap v228: 97 pgs: 9 creating+peering, 18 unknown, 70 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:24:00.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:00 smithi047 ceph-mon[21846]: mgrmap e20: smithi047.uutpnj(active, since 5m), standbys: smithi113.qvxssq 2024-04-22T22:24:00.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:00 smithi113 ceph-mon[24083]: pgmap v228: 97 pgs: 9 creating+peering, 18 unknown, 70 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:24:00.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:00 smithi113 ceph-mon[24083]: mgrmap e20: smithi047.uutpnj(active, since 5m), standbys: smithi113.qvxssq 2024-04-22T22:24:01.360 INFO:teuthology.orchestra.run.smithi047.stderr:Invalid command: Unexpected argument '--clusterid' 2024-04-22T22:24:01.360 INFO:teuthology.orchestra.run.smithi047.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-04-22T22:24:01.360 INFO:teuthology.orchestra.run.smithi047.stderr:Error EINVAL: invalid command 2024-04-22T22:24:01.719 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-04-22T22:24:01.720 INFO:teuthology.orchestra.run.smithi047.stdout: "bind": "/fake", 2024-04-22T22:24:01.720 INFO:teuthology.orchestra.run.smithi047.stdout: "fs": "foofs", 2024-04-22T22:24:01.720 INFO:teuthology.orchestra.run.smithi047.stdout: "path": "/", 2024-04-22T22:24:01.720 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster": "foo", 2024-04-22T22:24:01.720 INFO:teuthology.orchestra.run.smithi047.stdout: "mode": "RW" 2024-04-22T22:24:01.720 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-04-22T22:24:02.534 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-04-22T22:24:02.847 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: pgmap v229: 97 pgs: 9 creating+peering, 1 unknown, 87 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-22T22:24:02.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:02.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:02.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-22T22:24:02.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: Creating key for client.nfs.foo.1.0.smithi113.getqum 2024-04-22T22:24:02.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi113.getqum", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-22T22:24:02.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-22T22:24:02.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi113.getqum", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-22T22:24:02.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-22T22:24:02.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-22T22:24:02.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-22T22:24:02.848 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:02 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:02.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: pgmap v229: 97 pgs: 9 creating+peering, 1 unknown, 87 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-22T22:24:02.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:02.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:02.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-22T22:24:02.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: Creating key for client.nfs.foo.1.0.smithi113.getqum 2024-04-22T22:24:02.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi113.getqum", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-22T22:24:02.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-22T22:24:02.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi113.getqum", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-22T22:24:02.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-22T22:24:02.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-22T22:24:02.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-22T22:24:02.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:02 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:04.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:04 smithi047 ceph-mon[21846]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 153 B/s rd, 307 B/s wr, 0 op/s 2024-04-22T22:24:04.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:04 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-22T22:24:04.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:04 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-22T22:24:04.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:04 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi113.getqum-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:24:04.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:04 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi113.getqum-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:24:04.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:04 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:04.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:04 smithi113 ceph-mon[24083]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 153 B/s rd, 307 B/s wr, 0 op/s 2024-04-22T22:24:04.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:04 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-22T22:24:04.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:04 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-22T22:24:04.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:04 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi113.getqum-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:24:04.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:04 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi113.getqum-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:24:04.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:04 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:05.973 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:05 smithi113 ceph-mon[24083]: from='client.14530 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:05.974 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:05 smithi113 ceph-mon[24083]: Rados config object exists: conf-nfs.foo 2024-04-22T22:24:05.974 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:05 smithi113 ceph-mon[24083]: Creating key for client.nfs.foo.1.0.smithi113.getqum-rgw 2024-04-22T22:24:05.974 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:05 smithi113 ceph-mon[24083]: Deploying daemon nfs.foo.1.0.smithi113.getqum on smithi113 2024-04-22T22:24:06.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:05 smithi047 ceph-mon[21846]: from='client.14530 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:06.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:05 smithi047 ceph-mon[21846]: Rados config object exists: conf-nfs.foo 2024-04-22T22:24:06.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:05 smithi047 ceph-mon[21846]: Creating key for client.nfs.foo.1.0.smithi113.getqum-rgw 2024-04-22T22:24:06.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:05 smithi047 ceph-mon[21846]: Deploying daemon nfs.foo.1.0.smithi113.getqum on smithi113 2024-04-22T22:24:06.942 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:06 smithi113 ceph-mon[24083]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-22T22:24:06.942 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:06 smithi113 ceph-mon[24083]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:06.989 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:06 smithi047 ceph-mon[21846]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-22T22:24:06.989 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:06 smithi047 ceph-mon[21846]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:07.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:07 smithi047 ceph-mon[21846]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:07.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:07 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:07.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:07 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:24:07.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:07 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:07.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:07 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:24:07.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:07 smithi113 ceph-mon[24083]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:07.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:07 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:07.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:07 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:24:07.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:07 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:07.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:07 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:24:08.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:08 smithi113 ceph-mon[24083]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-22T22:24:09.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:08 smithi047 ceph-mon[21846]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-22T22:24:09.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:09 smithi113 ceph-mon[24083]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:10.002 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:09 smithi047 ceph-mon[21846]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:11.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:10 smithi047 ceph-mon[21846]: pgmap v233: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-22T22:24:11.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:10 smithi047 ceph-mon[21846]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:11.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:11.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:10 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:11.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:10 smithi113 ceph-mon[24083]: pgmap v233: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-22T22:24:11.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:10 smithi113 ceph-mon[24083]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:11.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:11.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:10 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:11.893 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:11 smithi047 ceph-mon[21846]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:12.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:11 smithi113 ceph-mon[24083]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:12.313 INFO:teuthology.orchestra.run.smithi047.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-04-22T22:24:12.978 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-22T22:24:12.989 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi047.front.sepia.ceph.com 2024-04-22T22:24:12.989 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-04-22T22:24:13.019 INFO:teuthology.orchestra.run.smithi047.stderr:+ mkdir /mnt/foo 2024-04-22T22:24:13.039 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-04-22T22:24:13.104 INFO:teuthology.orchestra.run.smithi047.stderr:++ hostname 2024-04-22T22:24:13.105 INFO:teuthology.orchestra.run.smithi047.stderr:+ mount -t nfs smithi047:/fake /mnt/foo -o sync 2024-04-22T22:24:13.238 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:12 smithi047 ceph-mon[21846]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-22T22:24:13.238 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:13.238 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:13.239 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:13.239 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:24:13.239 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:13.239 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:24:13.239 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:13.239 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:12 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:13.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:12 smithi113 ceph-mon[24083]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-22T22:24:13.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:13.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:13.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:13.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:24:13.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:13.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:24:13.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:13.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:12 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:13.499 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-04-22T22:24:13.564 INFO:teuthology.orchestra.run.smithi047.stderr:+ echo test 2024-04-22T22:24:13.575 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-04-22T22:24:13.639 INFO:teuthology.orchestra.run.smithi047.stderr:+ sync 2024-04-22T22:24:13.878 INFO:teuthology.run_tasks:Running task parallel... 2024-04-22T22:24:13.887 INFO:teuthology.task.parallel:starting parallel... 2024-04-22T22:24:13.888 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T22:24:13.888 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T22:24:13.889 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi047.front.sepia.ceph.com 2024-04-22T22:24:13.889 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-22T22:24:13.889 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T22:24:13.889 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-22T22:24:13.890 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T22:24:13.890 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi047.front.sepia.ceph.com 2024-04-22T22:24:13.890 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-04-22T22:24:13.954 INFO:teuthology.orchestra.run.smithi047.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-22T22:24:13.954 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-22T22:24:13.954 INFO:teuthology.orchestra.run.smithi047.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-22T22:24:13.954 INFO:teuthology.orchestra.run.smithi047.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-22T22:24:14.055 INFO:teuthology.orchestra.run.smithi047.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-22T22:24:14.055 INFO:teuthology.orchestra.run.smithi047.stdout:releasing clients 2024-04-22T22:24:14.142 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:13 smithi047 ceph-mon[21846]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:14.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:13 smithi113 ceph-mon[24083]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:15.055 INFO:teuthology.orchestra.run.smithi047.stdout: 5 304 55.80 MB/sec warmup 1 sec latency 10.503 ms 2024-04-22T22:24:15.180 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:14 smithi047 ceph-mon[21846]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-22T22:24:15.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:14 smithi113 ceph-mon[24083]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-22T22:24:15.976 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-04-22T22:24:15.977 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-04-22T22:24:15.977 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-04-22T22:24:15.977 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-22T22:24:15.977 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:24:15.977 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:24:16.055 INFO:teuthology.orchestra.run.smithi047.stdout: 5 596 51.77 MB/sec warmup 2 sec latency 13.654 ms 2024-04-22T22:24:16.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-04-22T22:24:16.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-04-22T22:24:16.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-04-22T22:24:16.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-22T22:24:16.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:24:16.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:24:16.981 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:16 smithi047 ceph-mon[21846]: pgmap v236: 97 pgs: 97 active+clean; 8.1 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 650 KiB/s wr, 16 op/s 2024-04-22T22:24:16.981 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-04-22T22:24:16.981 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-04-22T22:24:16.981 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-04-22T22:24:16.981 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-04-22T22:24:16.981 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:16 smithi047 ceph-mon[21846]: osdmap e69: 8 total, 8 up, 8 in 2024-04-22T22:24:17.057 INFO:teuthology.orchestra.run.smithi047.stdout: 5 1313 40.45 MB/sec warmup 3 sec latency 10.949 ms 2024-04-22T22:24:17.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:16 smithi113 ceph-mon[24083]: pgmap v236: 97 pgs: 97 active+clean; 8.1 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 650 KiB/s wr, 16 op/s 2024-04-22T22:24:17.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:16 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-04-22T22:24:17.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:16 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-04-22T22:24:17.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:16 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-04-22T22:24:17.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:16 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-04-22T22:24:17.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:16 smithi113 ceph-mon[24083]: osdmap e69: 8 total, 8 up, 8 in 2024-04-22T22:24:17.986 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:17 smithi047 ceph-mon[21846]: osdmap e70: 8 total, 8 up, 8 in 2024-04-22T22:24:18.055 INFO:teuthology.orchestra.run.smithi047.stdout: 5 2437 35.22 MB/sec warmup 4 sec latency 12.776 ms 2024-04-22T22:24:18.093 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-22T22:24:18.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:17 smithi113 ceph-mon[24083]: osdmap e70: 8 total, 8 up, 8 in 2024-04-22T22:24:18.994 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:18 smithi047 ceph-mon[21846]: pgmap v239: 97 pgs: 1 peering, 96 active+clean; 70 MiB data, 96 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 9.1 MiB/s wr, 363 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T22:24:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: 5 3466 37.21 MB/sec warmup 5 sec latency 17.837 ms 2024-04-22T22:24:19.246 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:18 smithi113 ceph-mon[24083]: pgmap v239: 97 pgs: 1 peering, 96 active+clean; 70 MiB data, 96 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 9.1 MiB/s wr, 363 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T22:24:20.055 INFO:teuthology.orchestra.run.smithi047.stdout: 5 3856 33.17 MB/sec warmup 6 sec latency 94.982 ms 2024-04-22T22:24:21.033 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:20 smithi047 ceph-mon[21846]: pgmap v240: 97 pgs: 1 peering, 96 active+clean; 81 MiB data, 101 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s wr, 394 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T22:24:21.055 INFO:teuthology.orchestra.run.smithi047.stdout: 5 4400 29.03 MB/sec warmup 7 sec latency 26.066 ms 2024-04-22T22:24:21.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:20 smithi113 ceph-mon[24083]: pgmap v240: 97 pgs: 1 peering, 96 active+clean; 81 MiB data, 101 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s wr, 394 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T22:24:22.055 INFO:teuthology.orchestra.run.smithi047.stdout: 5 5080 28.24 MB/sec warmup 8 sec latency 33.553 ms 2024-04-22T22:24:23.043 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:22 smithi047 ceph-mon[21846]: pgmap v241: 97 pgs: 1 peering, 96 active+clean; 128 MiB data, 202 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 19 MiB/s wr, 1.06k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:23.055 INFO:teuthology.orchestra.run.smithi047.stdout: 5 5915 26.11 MB/sec warmup 9 sec latency 20.975 ms 2024-04-22T22:24:23.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:22 smithi113 ceph-mon[24083]: pgmap v241: 97 pgs: 1 peering, 96 active+clean; 128 MiB data, 202 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 19 MiB/s wr, 1.06k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:23.288 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-22T22:24:24.056 INFO:teuthology.orchestra.run.smithi047.stdout: 5 6534 26.20 MB/sec warmup 10 sec latency 17.114 ms 2024-04-22T22:24:25.056 INFO:teuthology.orchestra.run.smithi047.stdout: 5 7289 26.50 MB/sec warmup 11 sec latency 29.291 ms 2024-04-22T22:24:25.066 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:24 smithi047 ceph-mon[21846]: pgmap v242: 97 pgs: 97 active+clean; 130 MiB data, 232 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 19 MiB/s wr, 1.09k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:25.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:24 smithi113 ceph-mon[24083]: pgmap v242: 97 pgs: 97 active+clean; 130 MiB data, 232 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 19 MiB/s wr, 1.09k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:26.056 INFO:teuthology.orchestra.run.smithi047.stdout: 5 7629 24.69 MB/sec warmup 12 sec latency 46.638 ms 2024-04-22T22:24:27.056 INFO:teuthology.orchestra.run.smithi047.stdout: 5 8179 23.32 MB/sec warmup 13 sec latency 19.530 ms 2024-04-22T22:24:27.092 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:26 smithi047 ceph-mon[21846]: pgmap v243: 97 pgs: 97 active+clean; 132 MiB data, 250 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 12 MiB/s wr, 810 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:27.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:26 smithi113 ceph-mon[24083]: pgmap v243: 97 pgs: 97 active+clean; 132 MiB data, 250 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 12 MiB/s wr, 810 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:28.059 INFO:teuthology.orchestra.run.smithi047.stdout: 5 8794 23.02 MB/sec warmup 14 sec latency 37.854 ms 2024-04-22T22:24:28.296 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph mgr module enable nfs --force' 2024-04-22T22:24:29.059 INFO:teuthology.orchestra.run.smithi047.stdout: 5 9465 22.00 MB/sec warmup 15 sec latency 31.120 ms 2024-04-22T22:24:29.094 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:28 smithi047 ceph-mon[21846]: pgmap v244: 97 pgs: 97 active+clean; 139 MiB data, 330 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 14 MiB/s wr, 998 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:29.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:28 smithi113 ceph-mon[24083]: pgmap v244: 97 pgs: 97 active+clean; 139 MiB data, 330 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 14 MiB/s wr, 998 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:30.059 INFO:teuthology.orchestra.run.smithi047.stdout: 5 10079 22.24 MB/sec warmup 16 sec latency 23.294 ms 2024-04-22T22:24:31.059 INFO:teuthology.orchestra.run.smithi047.stdout: 5 10571 22.04 MB/sec warmup 17 sec latency 62.347 ms 2024-04-22T22:24:31.133 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:30 smithi047 ceph-mon[21846]: pgmap v245: 97 pgs: 97 active+clean; 138 MiB data, 330 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 11 MiB/s wr, 857 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:31.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:30 smithi113 ceph-mon[24083]: pgmap v245: 97 pgs: 97 active+clean; 138 MiB data, 330 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 11 MiB/s wr, 857 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:31.991 INFO:teuthology.orchestra.run.smithi047.stderr:module 'nfs' is already enabled 2024-04-22T22:24:32.059 INFO:teuthology.orchestra.run.smithi047.stdout: 5 11102 21.73 MB/sec warmup 18 sec latency 25.148 ms 2024-04-22T22:24:32.143 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:31 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2425510066' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-22T22:24:32.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:31 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2425510066' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-22T22:24:33.060 INFO:teuthology.orchestra.run.smithi047.stdout: 5 11571 20.89 MB/sec warmup 19 sec latency 25.974 ms 2024-04-22T22:24:33.146 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-22T22:24:33.158 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:32 smithi047 ceph-mon[21846]: pgmap v246: 97 pgs: 97 active+clean; 148 MiB data, 397 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 13 MiB/s wr, 1.14k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:33.158 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:32 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2425510066' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-22T22:24:33.158 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:32 smithi047 ceph-mon[21846]: mgrmap e21: smithi047.uutpnj(active, since 6m), standbys: smithi113.qvxssq 2024-04-22T22:24:33.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:32 smithi113 ceph-mon[24083]: pgmap v246: 97 pgs: 97 active+clean; 148 MiB data, 397 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 13 MiB/s wr, 1.14k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T22:24:33.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:32 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2425510066' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-22T22:24:33.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:32 smithi113 ceph-mon[24083]: mgrmap e21: smithi047.uutpnj(active, since 6m), standbys: smithi113.qvxssq 2024-04-22T22:24:34.060 INFO:teuthology.orchestra.run.smithi047.stdout: 5 12154 20.58 MB/sec warmup 20 sec latency 21.346 ms 2024-04-22T22:24:35.060 INFO:teuthology.orchestra.run.smithi047.stdout: 5 12889 20.14 MB/sec warmup 21 sec latency 37.403 ms 2024-04-22T22:24:35.191 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:34 smithi047 ceph-mon[21846]: pgmap v247: 97 pgs: 97 active+clean; 149 MiB data, 412 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 725 op/s 2024-04-22T22:24:35.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:34 smithi113 ceph-mon[24083]: pgmap v247: 97 pgs: 97 active+clean; 149 MiB data, 412 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 725 op/s 2024-04-22T22:24:36.060 INFO:teuthology.orchestra.run.smithi047.stdout: 5 13280 19.85 MB/sec warmup 22 sec latency 50.195 ms 2024-04-22T22:24:36.680 INFO:teuthology.orchestra.run.smithi047.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:24:36.998 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:36 smithi113 ceph-mon[24083]: pgmap v248: 97 pgs: 97 active+clean; 149 MiB data, 410 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 710 op/s 2024-04-22T22:24:36.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:36 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:36.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:36 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:24:36.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:36 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:36.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:36 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:24:37.060 INFO:teuthology.orchestra.run.smithi047.stdout: 5 13882 20.09 MB/sec warmup 23 sec latency 28.260 ms 2024-04-22T22:24:37.211 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:36 smithi047 ceph-mon[21846]: pgmap v248: 97 pgs: 97 active+clean; 149 MiB data, 410 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 710 op/s 2024-04-22T22:24:37.211 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:36 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:37.211 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:36 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:24:37.211 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:36 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:37.211 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:36 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:24:37.866 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T22:24:37.867 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi047.front.sepia.ceph.com 2024-04-22T22:24:37.867 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-22T22:24:38.060 INFO:teuthology.orchestra.run.smithi047.stdout: 5 14434 20.15 MB/sec warmup 24 sec latency 26.725 ms 2024-04-22T22:24:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:37 smithi047 ceph-mon[21846]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:37 smithi047 ceph-mon[21846]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:24:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:37 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:37 smithi113 ceph-mon[24083]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:37 smithi113 ceph-mon[24083]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:24:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:38.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:37 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:39.060 INFO:teuthology.orchestra.run.smithi047.stdout: 5 14851 19.54 MB/sec warmup 25 sec latency 24.062 ms 2024-04-22T22:24:39.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:38 smithi047 ceph-mon[21846]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:24:39.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:38 smithi047 ceph-mon[21846]: pgmap v249: 97 pgs: 97 active+clean; 151 MiB data, 462 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 9.4 MiB/s wr, 897 op/s 2024-04-22T22:24:39.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:38 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:39.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:38 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:24:39.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:38 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:39.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:38 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:24:39.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:38 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:39.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:38 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:39.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:38 smithi047 ceph-mon[21846]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-22T22:24:39.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:38 smithi113 ceph-mon[24083]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:24:39.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:38 smithi113 ceph-mon[24083]: pgmap v249: 97 pgs: 97 active+clean; 151 MiB data, 462 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 9.4 MiB/s wr, 897 op/s 2024-04-22T22:24:39.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:38 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:39.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:38 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:24:39.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:38 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:24:39.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:38 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:24:39.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:38 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:39.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:38 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:39.248 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:38 smithi113 ceph-mon[24083]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-22T22:24:40.060 INFO:teuthology.orchestra.run.smithi047.stdout: 5 15616 19.39 MB/sec warmup 26 sec latency 22.585 ms 2024-04-22T22:24:40.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:39 smithi047 ceph-mon[21846]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T22:24:40.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:39 smithi047 ceph-mon[21846]: pgmap v250: 97 pgs: 97 active+clean; 151 MiB data, 462 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 741 op/s 2024-04-22T22:24:40.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:39 smithi113 ceph-mon[24083]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T22:24:40.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:39 smithi113 ceph-mon[24083]: pgmap v250: 97 pgs: 97 active+clean; 151 MiB data, 462 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 741 op/s 2024-04-22T22:24:41.060 INFO:teuthology.orchestra.run.smithi047.stdout: 5 16424 19.20 MB/sec warmup 27 sec latency 46.668 ms 2024-04-22T22:24:42.016 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:42 smithi047 ceph-mon[21846]: pgmap v251: 97 pgs: 97 active+clean; 154 MiB data, 509 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 757 op/s 2024-04-22T22:24:42.060 INFO:teuthology.orchestra.run.smithi047.stdout: 5 16842 19.04 MB/sec warmup 28 sec latency 39.000 ms 2024-04-22T22:24:42.072 INFO:teuthology.orchestra.run.smithi047.stdout:true 2024-04-22T22:24:42.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:42 smithi113 ceph-mon[24083]: pgmap v251: 97 pgs: 97 active+clean; 154 MiB data, 509 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 757 op/s 2024-04-22T22:24:42.622 INFO:teuthology.orchestra.run.smithi047.stdout:"Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-22T22:24:43.061 INFO:teuthology.orchestra.run.smithi047.stdout: 5 17436 19.24 MB/sec warmup 29 sec latency 44.453 ms 2024-04-22T22:24:43.616 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch ps' 2024-04-22T22:24:44.036 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:44 smithi047 ceph-mon[21846]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:44.036 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:44 smithi047 ceph-mon[21846]: pgmap v252: 97 pgs: 97 active+clean; 170 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 7.8 MiB/s wr, 760 op/s 2024-04-22T22:24:44.036 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:44 smithi047 ceph-mon[21846]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:44.060 INFO:teuthology.orchestra.run.smithi047.stdout: 5 18022 19.38 MB/sec warmup 30 sec latency 47.737 ms 2024-04-22T22:24:44.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:44 smithi113 ceph-mon[24083]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:44.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:44 smithi113 ceph-mon[24083]: pgmap v252: 97 pgs: 97 active+clean; 170 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 7.8 MiB/s wr, 760 op/s 2024-04-22T22:24:44.247 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:44 smithi113 ceph-mon[24083]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:45.061 INFO:teuthology.orchestra.run.smithi047.stdout: 5 18509 18.86 MB/sec warmup 31 sec latency 22.769 ms 2024-04-22T22:24:46.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 18927 18.61 MB/sec warmup 32 sec latency 48.798 ms 2024-04-22T22:24:46.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:46 smithi047 ceph-mon[21846]: pgmap v253: 97 pgs: 97 active+clean; 169 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.5 MiB/s wr, 749 op/s 2024-04-22T22:24:46.071 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:46 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:46.248 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:46 smithi113 ceph-mon[24083]: pgmap v253: 97 pgs: 97 active+clean; 169 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 7.5 MiB/s wr, 749 op/s 2024-04-22T22:24:46.248 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:46 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:24:47.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 19557 18.47 MB/sec warmup 33 sec latency 57.884 ms 2024-04-22T22:24:47.458 INFO:teuthology.orchestra.run.smithi047.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T22:24:47.458 INFO:teuthology.orchestra.run.smithi047.stdout:alertmanager.smithi047 smithi047 *:9093,9094 running (5m) 35s ago 6m 24.2M - ba2b418f427c 11fa9d9fcc72 2024-04-22T22:24:47.458 INFO:teuthology.orchestra.run.smithi047.stdout:crash.smithi047 smithi047 running (6m) 35s ago 6m 6409k - 17.2.0 e1d6a67b021e 1b99a2e1b2ad 2024-04-22T22:24:47.458 INFO:teuthology.orchestra.run.smithi047.stdout:crash.smithi113 smithi113 running (6m) 37s ago 6m 6400k - 17.2.0 e1d6a67b021e a35f59de8f2d 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:grafana.smithi047 smithi047 *:3000 running (5m) 35s ago 6m 55.9M - 8.3.5 dad864ee21e9 968e836a6db0 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:mds.foofs.smithi047.avrnmr smithi047 running (67s) 35s ago 65s 16.0M - 17.2.0 e1d6a67b021e 615ca2d95ab1 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:mds.foofs.smithi113.rtxmqv smithi113 running (63s) 37s ago 63s 13.5M - 17.2.0 e1d6a67b021e 079f760063a5 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:mgr.smithi047.uutpnj smithi047 *:9283 running (8m) 35s ago 8m 460M - 17.2.0 e1d6a67b021e 42f8f60aedc0 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:mgr.smithi113.qvxssq smithi113 *:8443,9283 running (6m) 37s ago 6m 414M - 17.2.0 e1d6a67b021e 7cab77c0494d 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:mon.smithi047 smithi047 running (8m) 35s ago 8m 47.2M 2048M 17.2.0 e1d6a67b021e 96cda5acb635 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:mon.smithi113 smithi113 running (6m) 37s ago 6m 37.6M 2048M 17.2.0 e1d6a67b021e 82b94d1b7aaf 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:nfs.foo.0.0.smithi047.vfwjww smithi047 *:2049 running (46s) 35s ago 45s 51.1M - 4.0 e1d6a67b021e ea186e24046a 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:nfs.foo.1.0.smithi113.getqum smithi113 *:2049 running (40s) 37s ago 40s 55.0M - 4.0 e1d6a67b021e 7cafdc505c1b 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:node-exporter.smithi047 smithi047 *:9100 running (6m) 35s ago 6m 17.4M - 1dbe0e931976 618a9c0cf648 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:node-exporter.smithi113 smithi113 *:9100 running (6m) 37s ago 6m 17.2M - 1dbe0e931976 495bdeae71f6 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:osd.0 smithi047 running (5m) 35s ago 5m 43.2M 3217M 17.2.0 e1d6a67b021e 12042fdd6c37 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:osd.1 smithi047 running (5m) 35s ago 5m 45.5M 3217M 17.2.0 e1d6a67b021e 077c4bd9730e 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:osd.2 smithi047 running (4m) 35s ago 4m 39.7M 3217M 17.2.0 e1d6a67b021e 5a898b3c4e7b 2024-04-22T22:24:47.459 INFO:teuthology.orchestra.run.smithi047.stdout:osd.3 smithi047 running (4m) 35s ago 4m 40.6M 3217M 17.2.0 e1d6a67b021e 31f106431cce 2024-04-22T22:24:47.460 INFO:teuthology.orchestra.run.smithi047.stdout:osd.4 smithi113 running (3m) 37s ago 3m 40.5M 3984M 17.2.0 e1d6a67b021e 6f83763ef7dc 2024-04-22T22:24:47.460 INFO:teuthology.orchestra.run.smithi047.stdout:osd.5 smithi113 running (3m) 37s ago 3m 41.8M 3984M 17.2.0 e1d6a67b021e e7cc308c1735 2024-04-22T22:24:47.460 INFO:teuthology.orchestra.run.smithi047.stdout:osd.6 smithi113 running (2m) 37s ago 2m 42.3M 3984M 17.2.0 e1d6a67b021e 87b85a4c560e 2024-04-22T22:24:47.460 INFO:teuthology.orchestra.run.smithi047.stdout:osd.7 smithi113 running (2m) 37s ago 2m 39.4M 3984M 17.2.0 e1d6a67b021e 8ab7d8d39212 2024-04-22T22:24:47.460 INFO:teuthology.orchestra.run.smithi047.stdout:prometheus.smithi047 smithi047 *:9095 running (5m) 35s ago 5m 62.0M - 514e6a882f6e d2f72369ca5a 2024-04-22T22:24:48.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 20452 18.51 MB/sec warmup 34 sec latency 21.783 ms 2024-04-22T22:24:48.079 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:48 smithi047 ceph-mon[21846]: pgmap v254: 97 pgs: 97 active+clean; 176 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 961 op/s 2024-04-22T22:24:48.497 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:48 smithi113 ceph-mon[24083]: pgmap v254: 97 pgs: 97 active+clean; 176 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 961 op/s 2024-04-22T22:24:48.639 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions' 2024-04-22T22:24:49.047 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:49 smithi047 ceph-mon[21846]: from='client.24411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:49.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 21254 18.94 MB/sec warmup 35 sec latency 21.490 ms 2024-04-22T22:24:49.497 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:49 smithi113 ceph-mon[24083]: from='client.24411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:24:50.053 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:50 smithi047 ceph-mon[21846]: pgmap v255: 97 pgs: 97 active+clean; 177 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 650 op/s 2024-04-22T22:24:50.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 21802 18.85 MB/sec warmup 36 sec latency 24.738 ms 2024-04-22T22:24:50.497 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:50 smithi113 ceph-mon[24083]: pgmap v255: 97 pgs: 97 active+clean; 177 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 650 op/s 2024-04-22T22:24:51.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 22414 18.62 MB/sec warmup 37 sec latency 40.242 ms 2024-04-22T22:24:52.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 23100 18.56 MB/sec warmup 38 sec latency 42.328 ms 2024-04-22T22:24:52.073 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": { 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": { 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": { 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": { 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: "overall": { 2024-04-22T22:24:52.074 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-22T22:24:52.075 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-04-22T22:24:52.075 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-04-22T22:24:52.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:52 smithi047 ceph-mon[21846]: pgmap v256: 97 pgs: 97 active+clean; 182 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 8.0 MiB/s wr, 686 op/s 2024-04-22T22:24:52.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:52 smithi113 ceph-mon[24083]: pgmap v256: 97 pgs: 97 active+clean; 182 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 8.0 MiB/s wr, 686 op/s 2024-04-22T22:24:53.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 23706 18.29 MB/sec warmup 39 sec latency 50.001 ms 2024-04-22T22:24:53.107 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:53 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1744402835' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:24:53.354 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-22T22:24:53.497 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:53 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/1744402835' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:24:54.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 24202 18.39 MB/sec warmup 40 sec latency 60.972 ms 2024-04-22T22:24:54.124 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:54 smithi047 ceph-mon[21846]: pgmap v257: 97 pgs: 97 active+clean; 196 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-04-22T22:24:54.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:54 smithi113 ceph-mon[24083]: pgmap v257: 97 pgs: 97 active+clean; 196 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-04-22T22:24:55.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 24779 18.53 MB/sec warmup 41 sec latency 42.356 ms 2024-04-22T22:24:55.855 INFO:teuthology.orchestra.run.smithi047.stdout:wait for servicemap items w/ changing names to refresh 2024-04-22T22:24:56.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 25255 18.42 MB/sec warmup 42 sec latency 32.062 ms 2024-04-22T22:24:56.144 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:56 smithi047 ceph-mon[21846]: pgmap v258: 97 pgs: 97 active+clean; 195 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 8.1 MiB/s wr, 769 op/s 2024-04-22T22:24:56.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:56 smithi113 ceph-mon[24083]: pgmap v258: 97 pgs: 97 active+clean; 195 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 8.1 MiB/s wr, 769 op/s 2024-04-22T22:24:57.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 25796 18.11 MB/sec warmup 43 sec latency 22.639 ms 2024-04-22T22:24:57.829 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'sleep 60' 2024-04-22T22:24:58.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 26414 18.12 MB/sec warmup 44 sec latency 27.669 ms 2024-04-22T22:24:58.157 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:24:58 smithi047 ceph-mon[21846]: pgmap v259: 97 pgs: 97 active+clean; 196 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 10 MiB/s wr, 950 op/s 2024-04-22T22:24:58.497 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:24:58 smithi113 ceph-mon[24083]: pgmap v259: 97 pgs: 97 active+clean; 196 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 10 MiB/s wr, 950 op/s 2024-04-22T22:24:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 27219 17.96 MB/sec warmup 45 sec latency 31.800 ms 2024-04-22T22:25:00.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 27922 18.20 MB/sec warmup 46 sec latency 21.691 ms 2024-04-22T22:25:00.178 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:00 smithi047 ceph-mon[21846]: pgmap v260: 97 pgs: 97 active+clean; 200 MiB data, 802 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.3 MiB/s wr, 830 op/s 2024-04-22T22:25:00.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:00 smithi113 ceph-mon[24083]: pgmap v260: 97 pgs: 97 active+clean; 200 MiB data, 802 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.3 MiB/s wr, 830 op/s 2024-04-22T22:25:01.063 INFO:teuthology.orchestra.run.smithi047.stdout: 5 28564 18.36 MB/sec warmup 47 sec latency 30.240 ms 2024-04-22T22:25:02.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 29035 18.15 MB/sec warmup 48 sec latency 29.699 ms 2024-04-22T22:25:02.201 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:02 smithi047 ceph-mon[21846]: pgmap v261: 97 pgs: 97 active+clean; 202 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 9.0 MiB/s wr, 924 op/s 2024-04-22T22:25:02.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:02 smithi113 ceph-mon[24083]: pgmap v261: 97 pgs: 97 active+clean; 202 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 9.0 MiB/s wr, 924 op/s 2024-04-22T22:25:03.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 29540 17.97 MB/sec warmup 49 sec latency 32.822 ms 2024-04-22T22:25:04.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 30484 17.98 MB/sec warmup 50 sec latency 22.169 ms 2024-04-22T22:25:04.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:04 smithi047 ceph-mon[21846]: pgmap v262: 97 pgs: 97 active+clean; 208 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-04-22T22:25:04.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:04 smithi113 ceph-mon[24083]: pgmap v262: 97 pgs: 97 active+clean; 208 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-04-22T22:25:05.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 31364 18.20 MB/sec warmup 51 sec latency 18.002 ms 2024-04-22T22:25:06.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 32243 18.51 MB/sec warmup 52 sec latency 28.021 ms 2024-04-22T22:25:06.069 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:06 smithi047 ceph-mon[21846]: pgmap v263: 97 pgs: 97 active+clean; 208 MiB data, 874 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 640 op/s 2024-04-22T22:25:06.299 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:06 smithi113 ceph-mon[24083]: pgmap v263: 97 pgs: 97 active+clean; 208 MiB data, 874 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 640 op/s 2024-04-22T22:25:07.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 33035 18.40 MB/sec warmup 53 sec latency 28.407 ms 2024-04-22T22:25:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 34042 18.43 MB/sec warmup 54 sec latency 15.877 ms 2024-04-22T22:25:08.266 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:08 smithi047 ceph-mon[21846]: pgmap v264: 97 pgs: 97 active+clean; 217 MiB data, 948 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 10 MiB/s wr, 916 op/s 2024-04-22T22:25:08.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:08 smithi113 ceph-mon[24083]: pgmap v264: 97 pgs: 97 active+clean; 217 MiB data, 948 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 10 MiB/s wr, 916 op/s 2024-04-22T22:25:09.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 34970 18.68 MB/sec warmup 55 sec latency 18.523 ms 2024-04-22T22:25:10.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 35775 18.90 MB/sec warmup 56 sec latency 14.492 ms 2024-04-22T22:25:10.279 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:10 smithi047 ceph-mon[21846]: pgmap v265: 97 pgs: 97 active+clean; 217 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.2 MiB/s wr, 741 op/s 2024-04-22T22:25:10.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:10 smithi113 ceph-mon[24083]: pgmap v265: 97 pgs: 97 active+clean; 217 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.2 MiB/s wr, 741 op/s 2024-04-22T22:25:11.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 36424 18.69 MB/sec warmup 57 sec latency 16.708 ms 2024-04-22T22:25:12.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 37199 18.70 MB/sec warmup 58 sec latency 18.868 ms 2024-04-22T22:25:12.292 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:12 smithi047 ceph-mon[21846]: pgmap v266: 97 pgs: 97 active+clean; 221 MiB data, 991 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 777 op/s 2024-04-22T22:25:12.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:12 smithi113 ceph-mon[24083]: pgmap v266: 97 pgs: 97 active+clean; 221 MiB data, 991 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 777 op/s 2024-04-22T22:25:13.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 37979 18.66 MB/sec warmup 59 sec latency 30.972 ms 2024-04-22T22:25:14.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 38884 18.96 MB/sec warmup 60 sec latency 20.214 ms 2024-04-22T22:25:14.307 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:14 smithi047 ceph-mon[21846]: pgmap v267: 97 pgs: 97 active+clean; 239 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-22T22:25:14.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:14 smithi113 ceph-mon[24083]: pgmap v267: 97 pgs: 97 active+clean; 239 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-22T22:25:15.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 39549 18.96 MB/sec warmup 61 sec latency 19.645 ms 2024-04-22T22:25:16.064 INFO:teuthology.orchestra.run.smithi047.stdout: 5 40136 18.82 MB/sec warmup 62 sec latency 23.668 ms 2024-04-22T22:25:16.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:16 smithi047 ceph-mon[21846]: pgmap v268: 97 pgs: 97 active+clean; 237 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 901 op/s 2024-04-22T22:25:16.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:25:16.068 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:16 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:25:16.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:16 smithi113 ceph-mon[24083]: pgmap v268: 97 pgs: 97 active+clean; 237 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 901 op/s 2024-04-22T22:25:16.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:16 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:25:16.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:16 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:25:17.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 40906 18.81 MB/sec warmup 63 sec latency 31.218 ms 2024-04-22T22:25:18.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 41759 18.82 MB/sec warmup 64 sec latency 19.274 ms 2024-04-22T22:25:18.078 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:18 smithi047 ceph-mon[21846]: pgmap v269: 97 pgs: 97 active+clean; 240 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.05k op/s 2024-04-22T22:25:18.498 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:18 smithi113 ceph-mon[24083]: pgmap v269: 97 pgs: 97 active+clean; 240 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.05k op/s 2024-04-22T22:25:19.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 42659 19.09 MB/sec warmup 65 sec latency 31.586 ms 2024-04-22T22:25:19.585 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:19 smithi047 ceph-mon[21846]: pgmap v270: 97 pgs: 97 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 935 op/s 2024-04-22T22:25:19.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:19 smithi113 ceph-mon[24083]: pgmap v270: 97 pgs: 97 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 935 op/s 2024-04-22T22:25:20.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 43088 18.97 MB/sec warmup 66 sec latency 32.938 ms 2024-04-22T22:25:21.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 43499 18.75 MB/sec warmup 67 sec latency 29.677 ms 2024-04-22T22:25:21.686 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:21 smithi047 ceph-mon[21846]: pgmap v271: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.12k op/s 2024-04-22T22:25:21.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:21 smithi113 ceph-mon[24083]: pgmap v271: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.12k op/s 2024-04-22T22:25:22.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 43872 18.60 MB/sec warmup 68 sec latency 29.789 ms 2024-04-22T22:25:23.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 44347 18.51 MB/sec warmup 69 sec latency 37.431 ms 2024-04-22T22:25:23.455 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:23 smithi047 ceph-mon[21846]: pgmap v272: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-04-22T22:25:23.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:23 smithi113 ceph-mon[24083]: pgmap v272: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-04-22T22:25:24.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 45038 18.46 MB/sec warmup 70 sec latency 22.494 ms 2024-04-22T22:25:25.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 45642 18.52 MB/sec warmup 71 sec latency 35.529 ms 2024-04-22T22:25:25.502 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:25 smithi047 ceph-mon[21846]: pgmap v273: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 782 op/s 2024-04-22T22:25:25.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:25 smithi113 ceph-mon[24083]: pgmap v273: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 782 op/s 2024-04-22T22:25:26.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 46187 18.56 MB/sec warmup 72 sec latency 55.772 ms 2024-04-22T22:25:27.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 46570 18.45 MB/sec warmup 73 sec latency 32.667 ms 2024-04-22T22:25:27.534 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:27 smithi047 ceph-mon[21846]: pgmap v274: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-22T22:25:27.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:27 smithi113 ceph-mon[24083]: pgmap v274: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-22T22:25:28.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 47000 18.27 MB/sec warmup 74 sec latency 42.675 ms 2024-04-22T22:25:29.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 47637 18.20 MB/sec warmup 75 sec latency 29.710 ms 2024-04-22T22:25:29.565 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:29 smithi047 ceph-mon[21846]: pgmap v275: 97 pgs: 97 active+clean; 264 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.5 MiB/s wr, 879 op/s 2024-04-22T22:25:29.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:29 smithi113 ceph-mon[24083]: pgmap v275: 97 pgs: 97 active+clean; 264 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.5 MiB/s wr, 879 op/s 2024-04-22T22:25:30.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 48288 18.15 MB/sec warmup 76 sec latency 44.806 ms 2024-04-22T22:25:31.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 48964 18.16 MB/sec warmup 77 sec latency 20.750 ms 2024-04-22T22:25:31.587 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:31 smithi047 ceph-mon[21846]: pgmap v276: 97 pgs: 97 active+clean; 267 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.1 MiB/s wr, 765 op/s 2024-04-22T22:25:31.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:31 smithi113 ceph-mon[24083]: pgmap v276: 97 pgs: 97 active+clean; 267 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.1 MiB/s wr, 765 op/s 2024-04-22T22:25:32.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 49641 18.30 MB/sec warmup 78 sec latency 24.001 ms 2024-04-22T22:25:33.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 50249 18.28 MB/sec warmup 79 sec latency 35.698 ms 2024-04-22T22:25:33.597 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:33 smithi047 ceph-mon[21846]: pgmap v277: 97 pgs: 97 active+clean; 272 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.7 MiB/s wr, 775 op/s 2024-04-22T22:25:33.749 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:33 smithi113 ceph-mon[24083]: pgmap v277: 97 pgs: 97 active+clean; 272 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.7 MiB/s wr, 775 op/s 2024-04-22T22:25:34.065 INFO:teuthology.orchestra.run.smithi047.stdout: 5 50838 18.19 MB/sec warmup 80 sec latency 28.448 ms 2024-04-22T22:25:35.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 51557 18.15 MB/sec warmup 81 sec latency 26.049 ms 2024-04-22T22:25:35.620 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:35 smithi047 ceph-mon[21846]: pgmap v278: 97 pgs: 97 active+clean; 275 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.8 MiB/s wr, 650 op/s 2024-04-22T22:25:35.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:35 smithi113 ceph-mon[24083]: pgmap v278: 97 pgs: 97 active+clean; 275 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.8 MiB/s wr, 650 op/s 2024-04-22T22:25:36.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 52411 18.18 MB/sec warmup 82 sec latency 35.008 ms 2024-04-22T22:25:37.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 53120 18.34 MB/sec warmup 83 sec latency 23.758 ms 2024-04-22T22:25:37.632 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:37 smithi047 ceph-mon[21846]: pgmap v279: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 9.3 MiB/s wr, 880 op/s 2024-04-22T22:25:37.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:37 smithi113 ceph-mon[24083]: pgmap v279: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 9.3 MiB/s wr, 880 op/s 2024-04-22T22:25:38.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 53714 18.31 MB/sec warmup 84 sec latency 35.487 ms 2024-04-22T22:25:38.645 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:38 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:25:38.645 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:38 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:25:38.645 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:38 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:25:38.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:38 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:25:38.748 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:38 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:25:38.749 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:38 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:25:39.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 54140 18.20 MB/sec warmup 85 sec latency 41.718 ms 2024-04-22T22:25:39.652 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:39 smithi047 ceph-mon[21846]: pgmap v280: 97 pgs: 97 active+clean; 283 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 7.5 MiB/s wr, 715 op/s 2024-04-22T22:25:39.652 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:39 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:25:39.652 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:39 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:25:39.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:39 smithi113 ceph-mon[24083]: pgmap v280: 97 pgs: 97 active+clean; 283 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 7.5 MiB/s wr, 715 op/s 2024-04-22T22:25:39.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:39 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:25:39.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:39 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' 2024-04-22T22:25:40.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 54709 18.09 MB/sec warmup 86 sec latency 29.514 ms 2024-04-22T22:25:41.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 55241 18.04 MB/sec warmup 87 sec latency 23.751 ms 2024-04-22T22:25:41.855 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:41 smithi047 ceph-mon[21846]: pgmap v281: 97 pgs: 97 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 829 op/s 2024-04-22T22:25:41.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:41 smithi113 ceph-mon[24083]: pgmap v281: 97 pgs: 97 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 829 op/s 2024-04-22T22:25:42.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 55914 18.02 MB/sec warmup 88 sec latency 27.568 ms 2024-04-22T22:25:43.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 56460 18.08 MB/sec warmup 89 sec latency 22.196 ms 2024-04-22T22:25:43.618 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:43 smithi047 ceph-mon[21846]: pgmap v282: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 914 op/s 2024-04-22T22:25:43.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:43 smithi113 ceph-mon[24083]: pgmap v282: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 914 op/s 2024-04-22T22:25:44.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 57058 18.09 MB/sec warmup 90 sec latency 34.888 ms 2024-04-22T22:25:45.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 57540 18.01 MB/sec warmup 91 sec latency 25.789 ms 2024-04-22T22:25:45.640 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:45 smithi047 ceph-mon[21846]: pgmap v283: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 723 op/s 2024-04-22T22:25:45.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:45 smithi113 ceph-mon[24083]: pgmap v283: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 723 op/s 2024-04-22T22:25:46.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 58050 17.91 MB/sec warmup 92 sec latency 33.008 ms 2024-04-22T22:25:47.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 58505 17.82 MB/sec warmup 93 sec latency 51.761 ms 2024-04-22T22:25:47.652 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:47 smithi047 ceph-mon[21846]: pgmap v284: 97 pgs: 97 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-04-22T22:25:47.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:47 smithi113 ceph-mon[24083]: pgmap v284: 97 pgs: 97 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-04-22T22:25:48.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 59203 17.83 MB/sec warmup 94 sec latency 35.258 ms 2024-04-22T22:25:49.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 59733 17.83 MB/sec warmup 95 sec latency 67.928 ms 2024-04-22T22:25:49.688 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:49 smithi047 ceph-mon[21846]: pgmap v285: 97 pgs: 97 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.1 MiB/s wr, 776 op/s 2024-04-22T22:25:49.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:49 smithi113 ceph-mon[24083]: pgmap v285: 97 pgs: 97 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.1 MiB/s wr, 776 op/s 2024-04-22T22:25:50.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 60200 17.83 MB/sec warmup 96 sec latency 45.960 ms 2024-04-22T22:25:51.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 60735 17.81 MB/sec warmup 97 sec latency 39.700 ms 2024-04-22T22:25:51.703 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:51 smithi047 ceph-mon[21846]: pgmap v286: 97 pgs: 97 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.1 MiB/s wr, 753 op/s 2024-04-22T22:25:51.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:51 smithi113 ceph-mon[24083]: pgmap v286: 97 pgs: 97 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.1 MiB/s wr, 753 op/s 2024-04-22T22:25:52.066 INFO:teuthology.orchestra.run.smithi047.stdout: 5 61122 17.70 MB/sec warmup 98 sec latency 28.025 ms 2024-04-22T22:25:53.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 61703 17.62 MB/sec warmup 99 sec latency 22.217 ms 2024-04-22T22:25:53.732 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:53 smithi047 ceph-mon[21846]: pgmap v287: 97 pgs: 97 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 792 op/s 2024-04-22T22:25:53.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:53 smithi113 ceph-mon[24083]: pgmap v287: 97 pgs: 97 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 792 op/s 2024-04-22T22:25:54.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 62217 17.58 MB/sec warmup 100 sec latency 25.631 ms 2024-04-22T22:25:55.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 62934 17.58 MB/sec warmup 101 sec latency 30.028 ms 2024-04-22T22:25:55.742 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:55 smithi047 ceph-mon[21846]: pgmap v288: 97 pgs: 97 active+clean; 316 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.1 KiB/s rd, 7.2 MiB/s wr, 701 op/s 2024-04-22T22:25:55.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:55 smithi113 ceph-mon[24083]: pgmap v288: 97 pgs: 97 active+clean; 316 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.1 KiB/s rd, 7.2 MiB/s wr, 701 op/s 2024-04-22T22:25:56.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 63423 17.60 MB/sec warmup 102 sec latency 60.043 ms 2024-04-22T22:25:57.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 63994 17.62 MB/sec warmup 103 sec latency 85.534 ms 2024-04-22T22:25:57.756 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:57 smithi047 ceph-mon[21846]: pgmap v289: 97 pgs: 97 active+clean; 317 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 9.3 MiB/s wr, 926 op/s 2024-04-22T22:25:57.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:57 smithi113 ceph-mon[24083]: pgmap v289: 97 pgs: 97 active+clean; 317 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 9.3 MiB/s wr, 926 op/s 2024-04-22T22:25:58.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 64431 17.57 MB/sec warmup 104 sec latency 30.035 ms 2024-04-22T22:25:59.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 64887 17.49 MB/sec warmup 105 sec latency 26.697 ms 2024-04-22T22:25:59.793 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:25:59 smithi047 ceph-mon[21846]: pgmap v290: 97 pgs: 97 active+clean; 319 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 6.4 MiB/s wr, 591 op/s 2024-04-22T22:25:59.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:25:59 smithi113 ceph-mon[24083]: pgmap v290: 97 pgs: 97 active+clean; 319 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 6.4 MiB/s wr, 591 op/s 2024-04-22T22:26:00.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 65456 17.39 MB/sec warmup 106 sec latency 28.176 ms 2024-04-22T22:26:01.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 66059 17.43 MB/sec warmup 107 sec latency 23.300 ms 2024-04-22T22:26:01.705 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:01 smithi047 ceph-mon[21846]: pgmap v291: 97 pgs: 97 active+clean; 323 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 7.2 MiB/s wr, 698 op/s 2024-04-22T22:26:01.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:01 smithi113 ceph-mon[24083]: pgmap v291: 97 pgs: 97 active+clean; 323 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 7.2 MiB/s wr, 698 op/s 2024-04-22T22:26:02.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 66626 17.36 MB/sec warmup 108 sec latency 53.813 ms 2024-04-22T22:26:02.296 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch ps' 2024-04-22T22:26:03.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 67155 17.42 MB/sec warmup 109 sec latency 21.240 ms 2024-04-22T22:26:03.733 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:03 smithi047 ceph-mon[21846]: pgmap v292: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.4 MiB/s wr, 814 op/s 2024-04-22T22:26:03.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:03 smithi113 ceph-mon[24083]: pgmap v292: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.4 MiB/s wr, 814 op/s 2024-04-22T22:26:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 67648 17.37 MB/sec warmup 110 sec latency 67.426 ms 2024-04-22T22:26:05.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 68107 17.34 MB/sec warmup 111 sec latency 27.222 ms 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:alertmanager.smithi047 smithi047 *:9093,9094 running (6m) 113s ago 8m 24.2M - ba2b418f427c 11fa9d9fcc72 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:crash.smithi047 smithi047 running (8m) 113s ago 8m 6409k - 17.2.0 e1d6a67b021e 1b99a2e1b2ad 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:crash.smithi113 smithi113 running (7m) 115s ago 7m 6400k - 17.2.0 e1d6a67b021e a35f59de8f2d 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:grafana.smithi047 smithi047 *:3000 running (6m) 113s ago 8m 55.9M - 8.3.5 dad864ee21e9 968e836a6db0 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:mds.foofs.smithi047.avrnmr smithi047 running (2m) 113s ago 2m 16.0M - 17.2.0 e1d6a67b021e 615ca2d95ab1 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:mds.foofs.smithi113.rtxmqv smithi113 running (2m) 115s ago 2m 13.5M - 17.2.0 e1d6a67b021e 079f760063a5 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:mgr.smithi047.uutpnj smithi047 *:9283 running (9m) 113s ago 9m 460M - 17.2.0 e1d6a67b021e 42f8f60aedc0 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:mgr.smithi113.qvxssq smithi113 *:8443,9283 running (7m) 115s ago 7m 414M - 17.2.0 e1d6a67b021e 7cab77c0494d 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:mon.smithi047 smithi047 running (9m) 113s ago 9m 47.2M 2048M 17.2.0 e1d6a67b021e 96cda5acb635 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:mon.smithi113 smithi113 running (7m) 115s ago 7m 37.6M 2048M 17.2.0 e1d6a67b021e 82b94d1b7aaf 2024-04-22T22:26:05.339 INFO:teuthology.orchestra.run.smithi047.stdout:nfs.foo.0.0.smithi047.vfwjww smithi047 *:2049 running (2m) 113s ago 2m 51.1M - 4.0 e1d6a67b021e ea186e24046a 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:nfs.foo.1.0.smithi113.getqum smithi113 *:2049 running (118s) 115s ago 118s 55.0M - 4.0 e1d6a67b021e 7cafdc505c1b 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:node-exporter.smithi047 smithi047 *:9100 running (7m) 113s ago 7m 17.4M - 1dbe0e931976 618a9c0cf648 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:node-exporter.smithi113 smithi113 *:9100 running (7m) 115s ago 7m 17.2M - 1dbe0e931976 495bdeae71f6 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:osd.0 smithi047 running (6m) 113s ago 6m 43.2M 3217M 17.2.0 e1d6a67b021e 12042fdd6c37 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:osd.1 smithi047 running (6m) 113s ago 6m 45.5M 3217M 17.2.0 e1d6a67b021e 077c4bd9730e 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:osd.2 smithi047 running (5m) 113s ago 5m 39.7M 3217M 17.2.0 e1d6a67b021e 5a898b3c4e7b 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:osd.3 smithi047 running (5m) 113s ago 5m 40.6M 3217M 17.2.0 e1d6a67b021e 31f106431cce 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:osd.4 smithi113 running (5m) 115s ago 5m 40.5M 3984M 17.2.0 e1d6a67b021e 6f83763ef7dc 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:osd.5 smithi113 running (4m) 115s ago 4m 41.8M 3984M 17.2.0 e1d6a67b021e e7cc308c1735 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:osd.6 smithi113 running (4m) 115s ago 4m 42.3M 3984M 17.2.0 e1d6a67b021e 87b85a4c560e 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:osd.7 smithi113 running (3m) 115s ago 3m 39.4M 3984M 17.2.0 e1d6a67b021e 8ab7d8d39212 2024-04-22T22:26:05.340 INFO:teuthology.orchestra.run.smithi047.stdout:prometheus.smithi047 smithi047 *:9095 running (7m) 113s ago 7m 62.0M - 514e6a882f6e d2f72369ca5a 2024-04-22T22:26:05.743 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:05 smithi047 ceph-mon[21846]: pgmap v293: 97 pgs: 97 active+clean; 334 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.1 MiB/s wr, 821 op/s 2024-04-22T22:26:05.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:05 smithi113 ceph-mon[24083]: pgmap v293: 97 pgs: 97 active+clean; 334 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.1 MiB/s wr, 821 op/s 2024-04-22T22:26:06.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 68747 17.28 MB/sec warmup 112 sec latency 33.924 ms 2024-04-22T22:26:06.544 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch upgrade status' 2024-04-22T22:26:06.747 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:06 smithi047 ceph-mon[21846]: from='client.24419 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:26:06.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:06 smithi113 ceph-mon[24083]: from='client.24419 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:26:07.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 69371 17.28 MB/sec warmup 113 sec latency 26.268 ms 2024-04-22T22:26:07.751 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:07 smithi047 ceph-mon[21846]: pgmap v294: 97 pgs: 97 active+clean; 342 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 9.4 MiB/s wr, 1.00k op/s 2024-04-22T22:26:07.900 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:07 smithi113 ceph-mon[24083]: pgmap v294: 97 pgs: 97 active+clean; 342 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 9.4 MiB/s wr, 1.00k op/s 2024-04-22T22:26:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: 5 70130 17.27 MB/sec warmup 114 sec latency 28.517 ms 2024-04-22T22:26:09.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 70720 17.34 MB/sec warmup 115 sec latency 18.822 ms 2024-04-22T22:26:09.775 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:09 smithi047 ceph-mon[21846]: pgmap v295: 97 pgs: 97 active+clean; 345 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 792 op/s 2024-04-22T22:26:09.904 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-04-22T22:26:09.904 INFO:teuthology.orchestra.run.smithi047.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", 2024-04-22T22:26:09.904 INFO:teuthology.orchestra.run.smithi047.stdout: "in_progress": true, 2024-04-22T22:26:09.904 INFO:teuthology.orchestra.run.smithi047.stdout: "services_complete": [], 2024-04-22T22:26:09.904 INFO:teuthology.orchestra.run.smithi047.stdout: "progress": "", 2024-04-22T22:26:09.904 INFO:teuthology.orchestra.run.smithi047.stdout: "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-22T22:26:09.905 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-04-22T22:26:09.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:09 smithi113 ceph-mon[24083]: pgmap v295: 97 pgs: 97 active+clean; 345 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 792 op/s 2024-04-22T22:26:10.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 71326 17.36 MB/sec warmup 116 sec latency 18.804 ms 2024-04-22T22:26:10.686 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:10 smithi047 ceph-mon[21846]: from='client.24423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:26:10.833 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph health detail' 2024-04-22T22:26:10.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:10 smithi113 ceph-mon[24083]: from='client.24423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:26:11.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 71841 17.33 MB/sec warmup 117 sec latency 22.545 ms 2024-04-22T22:26:11.795 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:11 smithi047 ceph-mon[21846]: pgmap v296: 97 pgs: 97 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 7.8 MiB/s wr, 810 op/s 2024-04-22T22:26:11.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:11 smithi113 ceph-mon[24083]: pgmap v296: 97 pgs: 97 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 7.8 MiB/s wr, 810 op/s 2024-04-22T22:26:12.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 72398 17.22 MB/sec warmup 118 sec latency 40.633 ms 2024-04-22T22:26:13.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 72837 17.16 MB/sec warmup 119 sec latency 29.035 ms 2024-04-22T22:26:13.844 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:13 smithi047 ceph-mon[21846]: pgmap v297: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 913 op/s 2024-04-22T22:26:13.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:13 smithi113 ceph-mon[24083]: pgmap v297: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 913 op/s 2024-04-22T22:26:14.338 INFO:teuthology.orchestra.run.smithi047.stdout:HEALTH_WARN Upgrade: failed to pull target image 2024-04-22T22:26:14.339 INFO:teuthology.orchestra.run.smithi047.stdout:[WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T22:26:14.339 INFO:teuthology.orchestra.run.smithi047.stdout: host smithi047 `cephadm pull` failed: cephadm exited with an error code: 1, stderr: Pulling container image quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f... 2024-04-22T22:26:14.339 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 125 from /bin/podman pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:26:14.339 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f... 2024-04-22T22:26:14.339 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Error: initializing source docker://quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f: reading manifest 430e09df97c8fc7dc2b2ae424f68ed11366c540f in quay.ceph.io/ceph-ci/ceph: manifest unknown 2024-04-22T22:26:14.339 INFO:teuthology.orchestra.run.smithi047.stdout:ERROR: Failed command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:26:14.856 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:14 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/2699902697' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T22:26:14.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:14 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/2699902697' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T22:26:15.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 74130 20.16 MB/sec execute 1 sec latency 59.193 ms 2024-04-22T22:26:15.620 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions' 2024-04-22T22:26:15.862 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:15 smithi047 ceph-mon[21846]: pgmap v298: 97 pgs: 97 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 8.1 MiB/s wr, 815 op/s 2024-04-22T22:26:15.863 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:26:15.863 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:15 smithi047 ceph-mon[21846]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:26:15.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:15 smithi113 ceph-mon[24083]: pgmap v298: 97 pgs: 97 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 8.1 MiB/s wr, 815 op/s 2024-04-22T22:26:15.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:26:15.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:15 smithi113 ceph-mon[24083]: from='mgr.14206 172.21.15.47:0/1851392367' entity='mgr.smithi047.uutpnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.uutpnj/trash_purge_schedule"}]: dispatch 2024-04-22T22:26:16.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 74795 19.99 MB/sec execute 2 sec latency 20.424 ms 2024-04-22T22:26:17.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 75194 17.78 MB/sec execute 3 sec latency 25.005 ms 2024-04-22T22:26:17.880 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:17 smithi047 ceph-mon[21846]: pgmap v299: 97 pgs: 97 active+clean; 327 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 9.5 MiB/s wr, 878 op/s 2024-04-22T22:26:17.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:17 smithi113 ceph-mon[24083]: pgmap v299: 97 pgs: 97 active+clean; 327 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 9.5 MiB/s wr, 878 op/s 2024-04-22T22:26:18.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 75649 15.31 MB/sec execute 4 sec latency 30.986 ms 2024-04-22T22:26:19.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 76214 14.46 MB/sec execute 5 sec latency 42.710 ms 2024-04-22T22:26:19.589 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-04-22T22:26:19.589 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": { 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": { 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": { 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": { 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: "overall": { 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-04-22T22:26:19.590 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-04-22T22:26:19.896 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:19 smithi047 ceph-mon[21846]: pgmap v300: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 729 op/s 2024-04-22T22:26:19.896 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:19 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1214024051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:26:20.000 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:19 smithi113 ceph-mon[24083]: pgmap v300: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 729 op/s 2024-04-22T22:26:20.000 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:19 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/1214024051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:26:20.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 76875 15.31 MB/sec execute 6 sec latency 29.976 ms 2024-04-22T22:26:20.853 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-22T22:26:21.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 77527 15.66 MB/sec execute 7 sec latency 32.660 ms 2024-04-22T22:26:21.865 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:21 smithi047 ceph-mon[21846]: pgmap v301: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.2 MiB/s wr, 796 op/s 2024-04-22T22:26:21.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:21 smithi113 ceph-mon[24083]: pgmap v301: 97 pgs: 97 active+clean; 336 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.2 MiB/s wr, 796 op/s 2024-04-22T22:26:22.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 78031 15.97 MB/sec execute 8 sec latency 33.059 ms 2024-04-22T22:26:23.068 INFO:teuthology.orchestra.run.smithi047.stdout: 5 78578 16.02 MB/sec execute 9 sec latency 108.401 ms 2024-04-22T22:26:23.919 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:23 smithi047 ceph-mon[21846]: pgmap v302: 97 pgs: 97 active+clean; 333 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.8 MiB/s wr, 1.00k op/s 2024-04-22T22:26:23.999 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:23 smithi113 ceph-mon[24083]: pgmap v302: 97 pgs: 97 active+clean; 333 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.8 MiB/s wr, 1.00k op/s 2024-04-22T22:26:24.069 INFO:teuthology.orchestra.run.smithi047.stdout: 5 78939 15.51 MB/sec execute 10 sec latency 94.671 ms 2024-04-22T22:26:24.434 INFO:teuthology.orchestra.run.smithi047.stdout:true 2024-04-22T22:26:24.678 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:24 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/3914245845' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:26:25.000 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:24 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/3914245845' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:26:25.070 INFO:teuthology.orchestra.run.smithi047.stdout: 5 79474 14.54 MB/sec execute 11 sec latency 44.848 ms 2024-04-22T22:26:25.334 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-22T22:26:25.689 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:25 smithi047 ceph-mon[21846]: pgmap v303: 97 pgs: 97 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.6 MiB/s wr, 825 op/s 2024-04-22T22:26:26.000 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:25 smithi113 ceph-mon[24083]: pgmap v303: 97 pgs: 97 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.6 MiB/s wr, 825 op/s 2024-04-22T22:26:26.070 INFO:teuthology.orchestra.run.smithi047.stdout: 5 79947 14.48 MB/sec execute 12 sec latency 31.522 ms 2024-04-22T22:26:27.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 80587 14.75 MB/sec execute 13 sec latency 31.484 ms 2024-04-22T22:26:27.712 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:27 smithi047 ceph-mon[21846]: pgmap v304: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.8 MiB/s wr, 852 op/s 2024-04-22T22:26:28.000 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:27 smithi113 ceph-mon[24083]: pgmap v304: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.8 MiB/s wr, 852 op/s 2024-04-22T22:26:28.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 81219 15.03 MB/sec execute 14 sec latency 22.071 ms 2024-04-22T22:26:29.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 81835 15.27 MB/sec execute 15 sec latency 22.644 ms 2024-04-22T22:26:29.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:26:29.712 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-22T22:26:29.916 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4aaff49bb1d84fe084ea6fecb008f356 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-22T22:26:29.919 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T22:26:29.929 INFO:tasks.cephadm:Teardown begin 2024-04-22T22:26:29.930 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:26:29.935 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:29 smithi047 ceph-mon[21846]: pgmap v305: 97 pgs: 97 active+clean; 328 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 6.9 MiB/s wr, 649 op/s 2024-04-22T22:26:29.935 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:29 smithi047 ceph-mon[21846]: from='client.? 172.21.15.47:0/1053936949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:26:29.974 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:26:29.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:29 smithi113 ceph-mon[24083]: pgmap v305: 97 pgs: 97 active+clean; 328 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 6.9 MiB/s wr, 649 op/s 2024-04-22T22:26:29.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:29 smithi113 ceph-mon[24083]: from='client.? 172.21.15.47:0/1053936949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:26:30.002 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T22:26:30.003 DEBUG:teuthology.orchestra.run.smithi047:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T22:26:30.035 DEBUG:teuthology.orchestra.run.smithi113:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T22:26:30.058 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T22:26:30.059 INFO:tasks.cephadm.mon.smithi047:Stopping mon.smithi047... 2024-04-22T22:26:30.059 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mon.smithi047 2024-04-22T22:26:30.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 82264 15.22 MB/sec execute 16 sec latency 33.805 ms 2024-04-22T22:26:30.247 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:30 smithi047 systemd[1]: Stopping Ceph mon.smithi047 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:26:30.247 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:30 smithi047 bash[54856]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon.smithi047" found: no such container 2024-04-22T22:26:30.498 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:30 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047[21841]: 2024-04-22T22:26:30.454+0000 7f3a82aa4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi047 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:26:30.498 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:30 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047[21841]: 2024-04-22T22:26:30.454+0000 7f3a82aa4700 -1 mon.smithi047@0(leader) e2 *** Got Signal Terminated *** 2024-04-22T22:26:30.749 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:30 smithi047 podman[54865]: 2024-04-22 22:26:30.507799533 +0000 UTC m=+0.297687561 container died 96cda5acb6353787a18bdf57e2fadf57f04f565897236605ed28e7993f67458e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047, release=754, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc., version=8, com.redhat.component=centos-stream-container, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , distribution-scope=public) 2024-04-22T22:26:31.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 82680 14.85 MB/sec execute 17 sec latency 34.618 ms 2024-04-22T22:26:31.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:31 smithi047 podman[54865]: 2024-04-22 22:26:31.222566394 +0000 UTC m=+1.012454409 container cleanup 96cda5acb6353787a18bdf57e2fadf57f04f565897236605ed28e7993f67458e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, release=754, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T22:26:31.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:31 smithi047 bash[54865]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi047 2024-04-22T22:26:31.260 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Apr 22 22:26:31 smithi047 bash[54887]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon.smithi047" found: no such container 2024-04-22T22:26:31.610 DEBUG:teuthology.orchestra.run.smithi047:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mon.smithi047.service' 2024-04-22T22:26:31.700 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:26:31.700 INFO:tasks.cephadm.mon.smithi047:Stopped mon.smithi047 2024-04-22T22:26:31.700 INFO:tasks.cephadm.mon.smithi113:Stopping mon.smithi113... 2024-04-22T22:26:31.700 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mon.smithi113 2024-04-22T22:26:31.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:31 smithi113 systemd[1]: Stopping Ceph mon.smithi113 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:26:31.995 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:31 smithi113 bash[40219]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon.smithi113" found: no such container 2024-04-22T22:26:31.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:31 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi113[24077]: 2024-04-22T22:26:31.940+0000 7f36676a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi113 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:26:31.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:31 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi113[24077]: 2024-04-22T22:26:31.940+0000 7f36676a4700 -1 mon.smithi113@1(peon) e2 *** Got Signal Terminated *** 2024-04-22T22:26:32.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83319 14.57 MB/sec execute 18 sec latency 123.400 ms 2024-04-22T22:26:32.250 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:32 smithi113 podman[40231]: 2024-04-22 22:26:32.003041782 +0000 UTC m=+0.218044408 container died 82b94d1b7aafd88134458827c0a91bb026cde160e5f0c5065e781db4635eb37c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi113, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vcs-type=git, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, name=centos-stream, GIT_CLEAN=True, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, distribution-scope=public, release=754, version=8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T22:26:32.750 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:32 smithi113 podman[40231]: 2024-04-22 22:26:32.550925603 +0000 UTC m=+0.765928232 container cleanup 82b94d1b7aafd88134458827c0a91bb026cde160e5f0c5065e781db4635eb37c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi113, distribution-scope=public, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, release=754, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , ceph=True, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-22T22:26:32.750 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:32 smithi113 bash[40231]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon-smithi113 2024-04-22T22:26:32.750 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 22 22:26:32 smithi113 bash[40257]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-mon.smithi113" found: no such container 2024-04-22T22:26:32.850 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@mon.smithi113.service' 2024-04-22T22:26:32.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:26:32.908 INFO:tasks.cephadm.mon.smithi113:Stopped mon.smithi113 2024-04-22T22:26:32.908 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T22:26:32.908 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.0 2024-04-22T22:26:33.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 13.88 MB/sec execute 19 sec latency 1123.449 ms 2024-04-22T22:26:33.396 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:32 smithi047 systemd[1]: Stopping Ceph osd.0 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:26:33.436 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:33 smithi047 bash[54937]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-22T22:26:33.437 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:33 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0[32288]: 2024-04-22T22:26:33.115+0000 7f896ecb9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:26:33.437 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:33 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0[32288]: 2024-04-22T22:26:33.115+0000 7f896ecb9700 -1 osd.0 70 *** Got signal Terminated *** 2024-04-22T22:26:33.437 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:33 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0[32288]: 2024-04-22T22:26:33.115+0000 7f896ecb9700 -1 osd.0 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:26:34.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 13.19 MB/sec execute 20 sec latency 2123.477 ms 2024-04-22T22:26:35.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 12.56 MB/sec execute 21 sec latency 3123.524 ms 2024-04-22T22:26:36.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 11.99 MB/sec execute 22 sec latency 4123.564 ms 2024-04-22T22:26:37.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 11.47 MB/sec execute 23 sec latency 5123.611 ms 2024-04-22T22:26:38.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 10.99 MB/sec execute 24 sec latency 6123.657 ms 2024-04-22T22:26:39.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 10.55 MB/sec execute 25 sec latency 7123.703 ms 2024-04-22T22:26:39.132 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:38 smithi047 podman[54950]: 2024-04-22 22:26:38.694423382 +0000 UTC m=+5.686602933 container died 12042fdd6c37c86ffb2e6875117f8c99827983cdc12e65bafcf29d4fc35fb63d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , architecture=x86_64, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, release=754, vcs-type=git, GIT_BRANCH=HEAD, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, name=centos-stream, distribution-scope=public) 2024-04-22T22:26:39.396 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:39 smithi047 podman[54950]: 2024-04-22 22:26:39.219094393 +0000 UTC m=+6.211273941 container cleanup 12042fdd6c37c86ffb2e6875117f8c99827983cdc12e65bafcf29d4fc35fb63d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, release=754, version=8, architecture=x86_64, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_BRANCH=HEAD) 2024-04-22T22:26:39.396 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:39 smithi047 bash[54950]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0 2024-04-22T22:26:39.701 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:39 smithi047 podman[54968]: 2024-04-22 22:26:39.439628644 +0000 UTC m=+0.738715110 container remove 12042fdd6c37c86ffb2e6875117f8c99827983cdc12e65bafcf29d4fc35fb63d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, distribution-scope=public, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.buildah.version=1.19.8, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-22T22:26:39.702 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:39 smithi047 bash[54981]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-22T22:26:39.987 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:39 smithi047 podman[55043]: 2024-04-22 22:26:39.613666047 +0000 UTC m=+0.019795364 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:26:39.987 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:39 smithi047 podman[55043]: 2024-04-22 22:26:39.729812465 +0000 UTC m=+0.135941776 container create b146ecab652c9e9dc6090088c2a91fb10b768fd81d4c18689310b730cdb425b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0-deactivate, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, vcs-type=git, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, release=754, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., name=centos-stream) 2024-04-22T22:26:39.987 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:39 smithi047 podman[55043]: 2024-04-22 22:26:39.990570017 +0000 UTC m=+0.396699321 container init b146ecab652c9e9dc6090088c2a91fb10b768fd81d4c18689310b730cdb425b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0-deactivate, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, architecture=x86_64, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, RELEASE=HEAD) 2024-04-22T22:26:40.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 10.15 MB/sec execute 26 sec latency 8123.738 ms 2024-04-22T22:26:40.354 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:39 smithi047 podman[55043]: 2024-04-22 22:26:39.999574072 +0000 UTC m=+0.405703375 container start b146ecab652c9e9dc6090088c2a91fb10b768fd81d4c18689310b730cdb425b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_CLEAN=True, release=754, com.redhat.component=centos-stream-container, distribution-scope=public, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.openshift.expose-services=, vcs-type=git) 2024-04-22T22:26:40.354 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:40 smithi047 podman[55043]: 2024-04-22 22:26:40.098976359 +0000 UTC m=+0.505105691 container attach b146ecab652c9e9dc6090088c2a91fb10b768fd81d4c18689310b730cdb425b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, GIT_CLEAN=True, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, version=8, release=754, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T22:26:40.646 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:40 smithi047 podman[55043]: 2024-04-22 22:26:40.362224063 +0000 UTC m=+0.768353373 container died b146ecab652c9e9dc6090088c2a91fb10b768fd81d4c18689310b730cdb425b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0-deactivate, vendor=Red Hat, Inc., vcs-type=git, release=754, io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, architecture=x86_64, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , version=8, ceph=True) 2024-04-22T22:26:40.996 INFO:journalctl@ceph.osd.0.smithi047.stdout:Apr 22 22:26:40 smithi047 podman[55069]: 2024-04-22 22:26:40.969352933 +0000 UTC m=+0.601056676 container remove b146ecab652c9e9dc6090088c2a91fb10b768fd81d4c18689310b730cdb425b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-0-deactivate, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., GIT_CLEAN=True, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_BRANCH=HEAD, io.openshift.expose-services=, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, architecture=x86_64, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T22:26:41.001 DEBUG:teuthology.orchestra.run.smithi047:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T22:26:41.039 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:26:41.039 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T22:26:41.039 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T22:26:41.039 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.1 2024-04-22T22:26:41.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 9.77 MB/sec execute 27 sec latency 9123.788 ms 2024-04-22T22:26:41.268 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:41 smithi047 systemd[1]: Stopping Ceph osd.1 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:26:41.269 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:41 smithi047 bash[55122]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-22T22:26:41.646 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:41 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1[35133]: 2024-04-22T22:26:41.274+0000 7f28378c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:26:41.647 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:41 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1[35133]: 2024-04-22T22:26:41.274+0000 7f28378c2700 -1 osd.1 70 *** Got signal Terminated *** 2024-04-22T22:26:41.647 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:41 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1[35133]: 2024-04-22T22:26:41.274+0000 7f28378c2700 -1 osd.1 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:26:42.071 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 9.42 MB/sec execute 28 sec latency 10123.834 ms 2024-04-22T22:26:43.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 9.10 MB/sec execute 29 sec latency 11123.902 ms 2024-04-22T22:26:44.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 8.79 MB/sec execute 30 sec latency 12123.948 ms 2024-04-22T22:26:45.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 8.51 MB/sec execute 31 sec latency 13123.974 ms 2024-04-22T22:26:46.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 8.24 MB/sec execute 32 sec latency 14124.024 ms 2024-04-22T22:26:47.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 7.99 MB/sec execute 33 sec latency 15124.050 ms 2024-04-22T22:26:47.098 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:46 smithi047 podman[55135]: 2024-04-22 22:26:46.834466064 +0000 UTC m=+5.655808932 container died 077c4bd9730edf62ff8465a5275fabfca9b00f634561664b8d87d9692aa5cdc2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, architecture=x86_64, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, name=centos-stream, ceph=True, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T22:26:47.396 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:47 smithi047 podman[55135]: 2024-04-22 22:26:47.19674463 +0000 UTC m=+6.018087497 container cleanup 077c4bd9730edf62ff8465a5275fabfca9b00f634561664b8d87d9692aa5cdc2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, name=centos-stream, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, release=754, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T22:26:47.397 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:47 smithi047 bash[55135]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1 2024-04-22T22:26:47.397 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:47 smithi047 bash[55166]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-22T22:26:47.803 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:47 smithi047 podman[55154]: 2024-04-22 22:26:47.456417231 +0000 UTC m=+0.623421850 container remove 077c4bd9730edf62ff8465a5275fabfca9b00f634561664b8d87d9692aa5cdc2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, version=8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , vcs-type=git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, distribution-scope=public, release=754, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T22:26:47.804 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:47 smithi047 podman[55230]: 2024-04-22 22:26:47.558260588 +0000 UTC m=+0.019554357 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:26:47.804 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:47 smithi047 podman[55230]: 2024-04-22 22:26:47.665539681 +0000 UTC m=+0.126833452 container create d9f0161526104315065ee51c6a9a7e2868f18d67234f38368a0bbd0b12c298a2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, version=8, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vcs-type=git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vendor=Red Hat, Inc.) 2024-04-22T22:26:48.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 7.76 MB/sec execute 34 sec latency 16124.096 ms 2024-04-22T22:26:48.146 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:47 smithi047 podman[55230]: 2024-04-22 22:26:47.926302428 +0000 UTC m=+0.387596194 container init d9f0161526104315065ee51c6a9a7e2868f18d67234f38368a0bbd0b12c298a2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, version=8, ceph=True, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, name=centos-stream, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, architecture=x86_64) 2024-04-22T22:26:48.147 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:47 smithi047 podman[55230]: 2024-04-22 22:26:47.935114744 +0000 UTC m=+0.396408519 container start d9f0161526104315065ee51c6a9a7e2868f18d67234f38368a0bbd0b12c298a2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1-deactivate, io.openshift.tags=base centos centos-stream, name=centos-stream, com.redhat.component=centos-stream-container, version=8, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, ceph=True, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-22T22:26:48.147 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:48 smithi047 podman[55230]: 2024-04-22 22:26:48.017894376 +0000 UTC m=+0.479188158 container attach d9f0161526104315065ee51c6a9a7e2868f18d67234f38368a0bbd0b12c298a2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1-deactivate, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, distribution-scope=public, version=8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.expose-services=) 2024-04-22T22:26:48.589 INFO:journalctl@ceph.osd.1.smithi047.stdout:Apr 22 22:26:48 smithi047 podman[55230]: 2024-04-22 22:26:48.289502095 +0000 UTC m=+0.750795867 container died d9f0161526104315065ee51c6a9a7e2868f18d67234f38368a0bbd0b12c298a2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-1-deactivate, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, ceph=True, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.component=centos-stream-container, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, release=754, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:26:48.913 DEBUG:teuthology.orchestra.run.smithi047:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T22:26:48.952 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:26:48.952 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T22:26:48.952 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T22:26:48.952 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.2 2024-04-22T22:26:49.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 7.54 MB/sec execute 35 sec latency 17124.126 ms 2024-04-22T22:26:49.146 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:49 smithi047 systemd[1]: Stopping Ceph osd.2 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:26:49.146 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:49 smithi047 bash[55308]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-22T22:26:49.646 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:49 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2[39009]: 2024-04-22T22:26:49.233+0000 7f83457d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:26:49.647 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:49 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2[39009]: 2024-04-22T22:26:49.233+0000 7f83457d0700 -1 osd.2 70 *** Got signal Terminated *** 2024-04-22T22:26:49.647 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:49 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2[39009]: 2024-04-22T22:26:49.233+0000 7f83457d0700 -1 osd.2 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:26:50.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 7.33 MB/sec execute 36 sec latency 18124.427 ms 2024-04-22T22:26:51.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 7.13 MB/sec execute 37 sec latency 19124.473 ms 2024-04-22T22:26:52.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 6.94 MB/sec execute 38 sec latency 20124.524 ms 2024-04-22T22:26:53.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 6.76 MB/sec execute 39 sec latency 21124.573 ms 2024-04-22T22:26:54.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 6.59 MB/sec execute 40 sec latency 22124.621 ms 2024-04-22T22:26:54.935 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:54 smithi047 podman[55320]: 2024-04-22 22:26:54.661222844 +0000 UTC m=+5.556193036 container died 5a898b3c4e7ba6858f758ca645a24f61e13d6047d27f3b1ce77538389c7671dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2, architecture=x86_64, release=754, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, version=8, name=centos-stream, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, ceph=True, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:26:55.072 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 6.43 MB/sec execute 41 sec latency 23124.647 ms 2024-04-22T22:26:55.301 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:55 smithi047 podman[55320]: 2024-04-22 22:26:55.065730998 +0000 UTC m=+5.960701203 container cleanup 5a898b3c4e7ba6858f758ca645a24f61e13d6047d27f3b1ce77538389c7671dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2, ceph=True, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, name=centos-stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, com.redhat.component=centos-stream-container, distribution-scope=public, version=8, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T22:26:55.301 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:55 smithi047 bash[55320]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2 2024-04-22T22:26:55.301 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:55 smithi047 bash[55351]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-22T22:26:55.638 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:55 smithi047 podman[55339]: 2024-04-22 22:26:55.308470014 +0000 UTC m=+0.641417752 container remove 5a898b3c4e7ba6858f758ca645a24f61e13d6047d27f3b1ce77538389c7671dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vcs-type=git, GIT_BRANCH=HEAD, version=8, io.openshift.expose-services=, distribution-scope=public) 2024-04-22T22:26:55.639 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:55 smithi047 podman[55419]: 2024-04-22 22:26:55.509461489 +0000 UTC m=+0.095004890 container create a7e952bed2201b897042e80a935b7992782f3fe4cce731ab8105fe19e92099cb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, release=754) 2024-04-22T22:26:55.639 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:55 smithi047 podman[55419]: 2024-04-22 22:26:55.433614537 +0000 UTC m=+0.019157959 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:26:55.897 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:55 smithi047 podman[55419]: 2024-04-22 22:26:55.728248952 +0000 UTC m=+0.313792354 container init a7e952bed2201b897042e80a935b7992782f3fe4cce731ab8105fe19e92099cb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2-deactivate, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, vcs-type=git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, release=754, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, ceph=True, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T22:26:55.897 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:55 smithi047 podman[55419]: 2024-04-22 22:26:55.738610236 +0000 UTC m=+0.324153638 container start a7e952bed2201b897042e80a935b7992782f3fe4cce731ab8105fe19e92099cb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., vcs-type=git, release=754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_CLEAN=True, version=8, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-22T22:26:55.897 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:55 smithi047 podman[55419]: 2024-04-22 22:26:55.811719281 +0000 UTC m=+0.397262691 container attach a7e952bed2201b897042e80a935b7992782f3fe4cce731ab8105fe19e92099cb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.buildah.version=1.19.8, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , architecture=x86_64, RELEASE=HEAD) 2024-04-22T22:26:56.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 6.28 MB/sec execute 42 sec latency 24124.677 ms 2024-04-22T22:26:56.383 INFO:journalctl@ceph.osd.2.smithi047.stdout:Apr 22 22:26:56 smithi047 podman[55419]: 2024-04-22 22:26:56.07757421 +0000 UTC m=+0.663117615 container died a7e952bed2201b897042e80a935b7992782f3fe4cce731ab8105fe19e92099cb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.19.8, GIT_CLEAN=True, distribution-scope=public, vendor=Red Hat, Inc., ceph=True, version=8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, architecture=x86_64, io.openshift.tags=base centos centos-stream, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T22:26:56.701 DEBUG:teuthology.orchestra.run.smithi047:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T22:26:56.755 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:26:56.755 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T22:26:56.755 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T22:26:56.756 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.3 2024-04-22T22:26:56.907 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:26:56 smithi047 systemd[1]: Stopping Ceph osd.3 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:26:57.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 6.13 MB/sec execute 43 sec latency 25124.724 ms 2024-04-22T22:26:57.397 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:26:56 smithi047 bash[55498]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-22T22:26:57.397 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:26:57 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3[42883]: 2024-04-22T22:26:57.043+0000 7f51e8bf2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:26:57.397 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:26:57 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3[42883]: 2024-04-22T22:26:57.043+0000 7f51e8bf2700 -1 osd.3 70 *** Got signal Terminated *** 2024-04-22T22:26:57.397 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:26:57 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3[42883]: 2024-04-22T22:26:57.043+0000 7f51e8bf2700 -1 osd.3 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:26:58.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 6.00 MB/sec execute 44 sec latency 26124.984 ms 2024-04-22T22:26:59.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 5.86 MB/sec execute 45 sec latency 27125.031 ms 2024-04-22T22:26:59.500 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:26:59 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:26:59.086+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:00.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 5.73 MB/sec execute 46 sec latency 28125.091 ms 2024-04-22T22:27:00.485 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:00 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:00.088+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:00.750 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:00 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:00.492+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:01.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 5.61 MB/sec execute 47 sec latency 29125.139 ms 2024-04-22T22:27:01.486 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:01 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:01.141+0000 7fa763316700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:37.423828+0000 front 2024-04-22T22:26:37.423874+0000 (oldest deadline 2024-04-22T22:27:00.324034+0000) 2024-04-22T22:27:01.486 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:01 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:01.053+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:01.750 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:01 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:01.492+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:01.896 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:01 smithi047 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3[42883]: 2024-04-22T22:27:01.580+0000 7f51e5d8e700 -1 osd.3 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:36.835579+0000 front 2024-04-22T22:26:36.835600+0000 (oldest deadline 2024-04-22T22:27:01.535712+0000) 2024-04-22T22:27:02.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 5.50 MB/sec execute 48 sec latency 30125.188 ms 2024-04-22T22:27:02.474 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:02 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:02.156+0000 7fa763316700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:37.423828+0000 front 2024-04-22T22:26:37.423874+0000 (oldest deadline 2024-04-22T22:27:00.324034+0000) 2024-04-22T22:27:02.474 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:02 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:02.049+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:02.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:02 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:02.480+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:02.751 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:02 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:02.636+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:02.787 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:02 smithi047 podman[55510]: 2024-04-22 22:27:02.521668052 +0000 UTC m=+5.597824551 container died 31f106431cce9ef8a08203169e5af8fe21f2b4145fcbc2b042603b14911c33b7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, release=754, io.buildah.version=1.19.8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:27:03.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 5.38 MB/sec execute 49 sec latency 31125.235 ms 2024-04-22T22:27:03.144 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:02 smithi047 podman[55510]: 2024-04-22 22:27:02.875908554 +0000 UTC m=+5.952065050 container cleanup 31f106431cce9ef8a08203169e5af8fe21f2b4145fcbc2b042603b14911c33b7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , ceph=True, architecture=x86_64, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, name=centos-stream, distribution-scope=public, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, release=754, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T22:27:03.145 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:02 smithi047 bash[55510]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3 2024-04-22T22:27:03.145 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:02 smithi047 bash[55540]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-22T22:27:03.397 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:03 smithi047 podman[55529]: 2024-04-22 22:27:03.151931431 +0000 UTC m=+0.621915999 container remove 31f106431cce9ef8a08203169e5af8fe21f2b4145fcbc2b042603b14911c33b7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3, version=8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, io.k8s.display-name=CentOS Stream 8, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:27:03.397 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:03 smithi047 podman[55606]: 2024-04-22 22:27:03.336007973 +0000 UTC m=+0.111401242 container create 18f6756e986974bb87fc2435806fb82f922f3b9bd154c5f576c75d34b82fb4a5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3-deactivate, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vendor=Red Hat, Inc., ceph=True, architecture=x86_64, name=centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T22:27:03.397 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:03 smithi047 podman[55606]: 2024-04-22 22:27:03.243469184 +0000 UTC m=+0.018862446 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:27:03.454 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:03 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:03.148+0000 7fa763316700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:37.423828+0000 front 2024-04-22T22:26:37.423874+0000 (oldest deadline 2024-04-22T22:27:00.324034+0000) 2024-04-22T22:27:03.454 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:03 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:03.074+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:03.750 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:03 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:03.668+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:03.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:03 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:03.460+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:03.897 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:03 smithi047 podman[55606]: 2024-04-22 22:27:03.588602287 +0000 UTC m=+0.363995551 container init 18f6756e986974bb87fc2435806fb82f922f3b9bd154c5f576c75d34b82fb4a5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, distribution-scope=public, GIT_CLEAN=True, ceph=True, com.redhat.component=centos-stream-container) 2024-04-22T22:27:03.897 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:03 smithi047 podman[55606]: 2024-04-22 22:27:03.596317352 +0000 UTC m=+0.371710621 container start 18f6756e986974bb87fc2435806fb82f922f3b9bd154c5f576c75d34b82fb4a5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vcs-type=git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, distribution-scope=public, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc.) 2024-04-22T22:27:03.897 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:03 smithi047 podman[55606]: 2024-04-22 22:27:03.655096173 +0000 UTC m=+0.430489443 container attach 18f6756e986974bb87fc2435806fb82f922f3b9bd154c5f576c75d34b82fb4a5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, distribution-scope=public, RELEASE=HEAD, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., release=754, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T22:27:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 5.28 MB/sec execute 50 sec latency 32125.282 ms 2024-04-22T22:27:04.260 INFO:journalctl@ceph.osd.3.smithi047.stdout:Apr 22 22:27:03 smithi047 podman[55606]: 2024-04-22 22:27:03.950665164 +0000 UTC m=+0.726058426 container died 18f6756e986974bb87fc2435806fb82f922f3b9bd154c5f576c75d34b82fb4a5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-3-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vcs-type=git, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, release=754, io.buildah.version=1.19.8, io.openshift.expose-services=, name=centos-stream) 2024-04-22T22:27:04.468 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:04 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:04.169+0000 7fa763316700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:37.423828+0000 front 2024-04-22T22:26:37.423874+0000 (oldest deadline 2024-04-22T22:27:00.324034+0000) 2024-04-22T22:27:04.468 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:04 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:04.052+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:04.606 DEBUG:teuthology.orchestra.run.smithi047:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T22:27:04.659 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:27:04.660 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T22:27:04.660 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T22:27:04.660 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.4 2024-04-22T22:27:04.727 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:04 smithi113 systemd[1]: Stopping Ceph osd.4 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:27:04.727 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:04 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:04.698+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:04.727 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:04 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:04.474+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:05.000 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:04 smithi113 bash[40310]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-22T22:27:05.001 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:04 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:04.917+0000 7fa76797d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:27:05.001 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:04 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:04.917+0000 7fa76797d700 -1 osd.4 70 *** Got signal Terminated *** 2024-04-22T22:27:05.001 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:04 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:04.917+0000 7fa76797d700 -1 osd.4 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:27:05.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 5.17 MB/sec execute 51 sec latency 33125.323 ms 2024-04-22T22:27:05.437 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:05 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:05.056+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:05.437 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:05 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:05.132+0000 7fa763316700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:37.423828+0000 front 2024-04-22T22:26:37.423874+0000 (oldest deadline 2024-04-22T22:27:00.324034+0000) 2024-04-22T22:27:05.750 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:05 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:05.679+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:05.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:05 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:05.443+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:06.073 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 5.07 MB/sec execute 52 sec latency 34125.363 ms 2024-04-22T22:27:06.400 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:06 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:06.088+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:06.400 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:06 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:06.150+0000 7fa763316700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:37.423828+0000 front 2024-04-22T22:26:37.423874+0000 (oldest deadline 2024-04-22T22:27:00.324034+0000) 2024-04-22T22:27:06.751 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:06 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:06.656+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:06.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:06 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:06.406+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:07.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.98 MB/sec execute 53 sec latency 35125.410 ms 2024-04-22T22:27:07.356 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:07 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:07.150+0000 7fa763316700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:37.423828+0000 front 2024-04-22T22:26:37.423874+0000 (oldest deadline 2024-04-22T22:27:00.324034+0000) 2024-04-22T22:27:07.357 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:07 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:07.055+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:07.636 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:07 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:07.363+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:08.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:07 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:07.643+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:08.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:07 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:07.643+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.612572+0000 front 2024-04-22T22:26:42.609348+0000 (oldest deadline 2024-04-22T22:27:07.309596+0000) 2024-04-22T22:27:08.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.89 MB/sec execute 54 sec latency 36125.458 ms 2024-04-22T22:27:08.363 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:08 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:08.044+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:08.363 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:08 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:08.172+0000 7fa763316700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:37.423828+0000 front 2024-04-22T22:26:37.423874+0000 (oldest deadline 2024-04-22T22:27:00.324034+0000) 2024-04-22T22:27:08.364 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:08 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:08.172+0000 7fa763316700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:45.624839+0000 front 2024-04-22T22:26:45.624940+0000 (oldest deadline 2024-04-22T22:27:07.324983+0000) 2024-04-22T22:27:08.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:08 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:08.369+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:08.751 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:08 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:08.609+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:08.751 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:08 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:08.609+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.612572+0000 front 2024-04-22T22:26:42.609348+0000 (oldest deadline 2024-04-22T22:27:07.309596+0000) 2024-04-22T22:27:09.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.80 MB/sec execute 55 sec latency 37125.503 ms 2024-04-22T22:27:09.251 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:09 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:09.135+0000 7fa763316700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:37.423828+0000 front 2024-04-22T22:26:37.423874+0000 (oldest deadline 2024-04-22T22:27:00.324034+0000) 2024-04-22T22:27:09.251 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:09 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4[28331]: 2024-04-22T22:27:09.135+0000 7fa763316700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:45.624839+0000 front 2024-04-22T22:26:45.624940+0000 (oldest deadline 2024-04-22T22:27:07.324983+0000) 2024-04-22T22:27:09.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:09 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:09.003+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:09.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:09 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:09.003+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:09.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:09 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:09.360+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:09.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:09 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:09.360+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:09.751 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:09 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:09.563+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:09.751 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:09 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:09.563+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.612572+0000 front 2024-04-22T22:26:42.609348+0000 (oldest deadline 2024-04-22T22:27:07.309596+0000) 2024-04-22T22:27:10.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.71 MB/sec execute 56 sec latency 38125.543 ms 2024-04-22T22:27:10.331 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:10 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:10.009+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:10.332 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:10 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:10.009+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:10.645 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:10 smithi113 podman[40323]: 2024-04-22 22:27:10.338501704 +0000 UTC m=+5.595420208 container died 6f83763ef7dc6869425476d18db865185fb4aad311a88007d36cf931902f788f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4, GIT_CLEAN=True, version=8, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, maintainer=Guillaume Abrioux , ceph=True, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.component=centos-stream-container, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, name=centos-stream) 2024-04-22T22:27:10.646 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:10 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:10.537+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:10.646 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:10 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:10.537+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.612572+0000 front 2024-04-22T22:26:42.609348+0000 (oldest deadline 2024-04-22T22:27:07.309596+0000) 2024-04-22T22:27:10.646 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:10 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:10.339+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:10.646 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:10 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:10.340+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:10.976 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:10 smithi113 podman[40323]: 2024-04-22 22:27:10.834005299 +0000 UTC m=+6.090923852 container cleanup 6f83763ef7dc6869425476d18db865185fb4aad311a88007d36cf931902f788f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, ceph=True, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-type=git, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, architecture=x86_64, io.openshift.tags=base centos centos-stream) 2024-04-22T22:27:10.976 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:10 smithi113 bash[40323]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4 2024-04-22T22:27:10.976 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:10 smithi113 bash[40353]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-22T22:27:11.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.63 MB/sec execute 57 sec latency 39125.581 ms 2024-04-22T22:27:11.251 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:11 smithi113 podman[40340]: 2024-04-22 22:27:11.102054406 +0000 UTC m=+0.756205023 container remove 6f83763ef7dc6869425476d18db865185fb4aad311a88007d36cf931902f788f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_BRANCH=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, distribution-scope=public, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-04-22T22:27:11.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:10 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:10.981+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:11.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:10 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:10.981+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:11.548 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:11 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:11.290+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:11.548 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:11 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:11.290+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:11.549 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:11 smithi113 podman[40414]: 2024-04-22 22:27:11.234960113 +0000 UTC m=+0.019086495 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:27:11.549 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:11 smithi113 podman[40414]: 2024-04-22 22:27:11.352839998 +0000 UTC m=+0.136966358 container create c7041ed9777a0e7a036a0b6e5b74677f0a78a84eeacd618835379d7a6ed62177 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , version=8, io.openshift.expose-services=, distribution-scope=public, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, vcs-type=git, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, architecture=x86_64, release=754, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870) 2024-04-22T22:27:11.549 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:11 smithi113 podman[40414]: 2024-04-22 22:27:11.555262596 +0000 UTC m=+0.339388956 container init c7041ed9777a0e7a036a0b6e5b74677f0a78a84eeacd618835379d7a6ed62177 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , version=8, ceph=True, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, name=centos-stream, release=754) 2024-04-22T22:27:11.832 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:11 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:11.574+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:11.832 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:11 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:11.575+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.612572+0000 front 2024-04-22T22:26:42.609348+0000 (oldest deadline 2024-04-22T22:27:07.309596+0000) 2024-04-22T22:27:11.833 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:11 smithi113 podman[40414]: 2024-04-22 22:27:11.558870107 +0000 UTC m=+0.342996473 container start c7041ed9777a0e7a036a0b6e5b74677f0a78a84eeacd618835379d7a6ed62177 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4-deactivate, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, com.redhat.component=centos-stream-container, vcs-type=git, version=8, io.openshift.expose-services=, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, name=centos-stream, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T22:27:11.833 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:11 smithi113 podman[40414]: 2024-04-22 22:27:11.671788329 +0000 UTC m=+0.455914698 container attach c7041ed9777a0e7a036a0b6e5b74677f0a78a84eeacd618835379d7a6ed62177 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, name=centos-stream) 2024-04-22T22:27:12.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.55 MB/sec execute 58 sec latency 40125.624 ms 2024-04-22T22:27:12.143 INFO:journalctl@ceph.osd.4.smithi113.stdout:Apr 22 22:27:11 smithi113 podman[40414]: 2024-04-22 22:27:11.83946508 +0000 UTC m=+0.623591435 container died c7041ed9777a0e7a036a0b6e5b74677f0a78a84eeacd618835379d7a6ed62177 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, release=754, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, GIT_BRANCH=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.k8s.display-name=CentOS Stream 8) 2024-04-22T22:27:12.143 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:11 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:11.987+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:12.143 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:11 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:11.987+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:12.460 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:12 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:12.267+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:12.460 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:12 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:12.267+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:12.478 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T22:27:12.519 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:27:12.520 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T22:27:12.520 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T22:27:12.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.5 2024-04-22T22:27:12.716 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:12 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:12.554+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:12.716 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:12 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:12.554+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.612572+0000 front 2024-04-22T22:26:42.609348+0000 (oldest deadline 2024-04-22T22:27:07.309596+0000) 2024-04-22T22:27:12.716 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:12 smithi113 systemd[1]: Stopping Ceph osd.5 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:27:12.717 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:12 smithi113 bash[40494]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-22T22:27:13.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:12 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:12.750+0000 7fcdc284a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:27:13.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:12 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:12.750+0000 7fcdc284a700 -1 osd.5 70 *** Got signal Terminated *** 2024-04-22T22:27:13.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:12 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:12.750+0000 7fcdc284a700 -1 osd.5 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:27:13.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.47 MB/sec execute 59 sec latency 41125.668 ms 2024-04-22T22:27:13.260 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:13 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:13.010+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:13.261 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:13 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:13.010+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:13.538 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:13 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:13.266+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:13.538 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:13 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:13.266+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:14.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:13 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:13.544+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:14.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:13 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:13.544+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.612572+0000 front 2024-04-22T22:26:42.609348+0000 (oldest deadline 2024-04-22T22:27:07.309596+0000) 2024-04-22T22:27:14.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.40 MB/sec execute 60 sec latency 42125.708 ms 2024-04-22T22:27:14.501 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:14 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:14.031+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:14.501 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:14 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:14.031+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:14.501 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:14 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:14.274+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:14.502 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:14 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:14.274+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:15.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:14 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:14.535+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:15.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:14 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:14.535+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.612572+0000 front 2024-04-22T22:26:42.609348+0000 (oldest deadline 2024-04-22T22:27:07.309596+0000) 2024-04-22T22:27:15.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.32 MB/sec execute 61 sec latency 43125.755 ms 2024-04-22T22:27:15.497 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:15 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:15.062+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:15.497 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:15 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:15.062+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:15.497 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:15 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:15.240+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:15.497 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:15 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:15.240+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:15.753 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:15 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:15.502+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:15.754 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:15 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:15.503+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.612572+0000 front 2024-04-22T22:26:42.609348+0000 (oldest deadline 2024-04-22T22:27:07.309596+0000) 2024-04-22T22:27:16.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.25 MB/sec execute 62 sec latency 44125.800 ms 2024-04-22T22:27:16.501 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:16 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:16.031+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:16.501 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:16 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:16.031+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:16.502 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:16 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:16.268+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:16.502 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:16 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:16.268+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:16.989 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:16 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:16.540+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:16.989 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:16 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:16.540+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.612572+0000 front 2024-04-22T22:26:42.609348+0000 (oldest deadline 2024-04-22T22:27:07.309596+0000) 2024-04-22T22:27:17.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.19 MB/sec execute 63 sec latency 45125.850 ms 2024-04-22T22:27:17.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:16 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:16.995+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:17.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:16 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:16.995+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:17.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:17 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:17.292+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:17.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:17 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:17.292+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:17.751 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:17 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:17.533+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:32.608190+0000 front 2024-04-22T22:26:32.611955+0000 (oldest deadline 2024-04-22T22:27:02.608945+0000) 2024-04-22T22:27:17.751 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:17 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:17.533+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.612572+0000 front 2024-04-22T22:26:42.609348+0000 (oldest deadline 2024-04-22T22:27:07.309596+0000) 2024-04-22T22:27:17.752 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:17 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5[31303]: 2024-04-22T22:27:17.533+0000 7fcdbd9e2700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.215704+0000 front 2024-04-22T22:26:53.210427+0000 (oldest deadline 2024-04-22T22:27:17.310533+0000) 2024-04-22T22:27:18.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.12 MB/sec execute 64 sec latency 46125.901 ms 2024-04-22T22:27:18.307 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:18 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:18.041+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:18.308 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:18 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:18.041+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:18.613 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:18 smithi113 podman[40507]: 2024-04-22 22:27:18.343176533 +0000 UTC m=+5.698903292 container died e7cc308c17356b0a00c02f8679ef68e82ab80ed5a6829a99f4e407671ba8b8f3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , release=754, name=centos-stream, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T22:27:18.614 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:18 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:18.312+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:18.614 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:18 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:18.312+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:18.614 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:18 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:18.312+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:19.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:18 smithi113 podman[40507]: 2024-04-22 22:27:18.819820825 +0000 UTC m=+6.175547593 container cleanup e7cc308c17356b0a00c02f8679ef68e82ab80ed5a6829a99f4e407671ba8b8f3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, vcs-type=git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, version=8, ceph=True, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream) 2024-04-22T22:27:19.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:18 smithi113 bash[40507]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5 2024-04-22T22:27:19.001 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:18 smithi113 bash[40548]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-22T22:27:19.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.06 MB/sec execute 65 sec latency 47125.947 ms 2024-04-22T22:27:19.311 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:19 smithi113 podman[40536]: 2024-04-22 22:27:19.129796132 +0000 UTC m=+0.782747151 container remove e7cc308c17356b0a00c02f8679ef68e82ab80ed5a6829a99f4e407671ba8b8f3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, name=centos-stream, RELEASE=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754) 2024-04-22T22:27:19.311 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:19 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:19.055+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:19.312 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:19 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:19.055+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:19.312 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:19 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:19.055+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:48.618376+0000 front 2024-04-22T22:26:48.618350+0000 (oldest deadline 2024-04-22T22:27:18.619008+0000) 2024-04-22T22:27:19.567 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:19 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:19.356+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:19.568 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:19 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:19.356+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:19.568 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:19 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:19.356+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:19.568 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:19 smithi113 podman[40613]: 2024-04-22 22:27:19.22143366 +0000 UTC m=+0.019650057 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:27:19.568 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:19 smithi113 podman[40613]: 2024-04-22 22:27:19.33883512 +0000 UTC m=+0.137051534 container create 06118e10f7177ca8eb0d4d29aa481f1d7d0d1dd898632a2d51d908213a404e0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vendor=Red Hat, Inc., ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, RELEASE=HEAD, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream) 2024-04-22T22:27:19.568 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:19 smithi113 podman[40613]: 2024-04-22 22:27:19.574715936 +0000 UTC m=+0.372932321 container init 06118e10f7177ca8eb0d4d29aa481f1d7d0d1dd898632a2d51d908213a404e0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5-deactivate, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_CLEAN=True, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., release=754, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.component=centos-stream-container) 2024-04-22T22:27:19.846 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:19 smithi113 podman[40613]: 2024-04-22 22:27:19.578005633 +0000 UTC m=+0.376222005 container start 06118e10f7177ca8eb0d4d29aa481f1d7d0d1dd898632a2d51d908213a404e0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, ceph=True) 2024-04-22T22:27:19.847 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:19 smithi113 podman[40613]: 2024-04-22 22:27:19.657920128 +0000 UTC m=+0.456136516 container attach 06118e10f7177ca8eb0d4d29aa481f1d7d0d1dd898632a2d51d908213a404e0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5-deactivate, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, name=centos-stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, RELEASE=HEAD, io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_CLEAN=True) 2024-04-22T22:27:20.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 4.00 MB/sec execute 66 sec latency 48125.967 ms 2024-04-22T22:27:20.162 INFO:journalctl@ceph.osd.5.smithi113.stdout:Apr 22 22:27:19 smithi113 podman[40613]: 2024-04-22 22:27:19.85393872 +0000 UTC m=+0.652155094 container died 06118e10f7177ca8eb0d4d29aa481f1d7d0d1dd898632a2d51d908213a404e0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-5-deactivate, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, io.k8s.display-name=CentOS Stream 8, version=8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:27:20.163 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:20 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:20.060+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:20.163 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:20 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:20.060+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:20.163 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:20 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:20.060+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:48.618376+0000 front 2024-04-22T22:26:48.618350+0000 (oldest deadline 2024-04-22T22:27:18.619008+0000) 2024-04-22T22:27:20.501 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:20 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:20.308+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:20.501 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:20 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:20.308+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:20.501 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:20 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:20.308+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:20.597 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T22:27:20.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:27:20.640 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T22:27:20.640 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-22T22:27:20.640 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.6 2024-04-22T22:27:20.908 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:20 smithi113 systemd[1]: Stopping Ceph osd.6 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:27:20.908 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:20 smithi113 bash[40693]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-22T22:27:20.908 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:20 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:20.913+0000 7fc6f318d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:27:21.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.94 MB/sec execute 67 sec latency 49126.011 ms 2024-04-22T22:27:21.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:20 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:20.913+0000 7fc6f318d700 -1 osd.6 70 *** Got signal Terminated *** 2024-04-22T22:27:21.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:20 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:20.913+0000 7fc6f318d700 -1 osd.6 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:27:21.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:21 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:21.071+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:21.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:21 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:21.071+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:21.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:21 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:21.071+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:48.618376+0000 front 2024-04-22T22:26:48.618350+0000 (oldest deadline 2024-04-22T22:27:18.619008+0000) 2024-04-22T22:27:21.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:21 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:21.354+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:21.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:21 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:21.354+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:21.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:21 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:21.354+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:22.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.88 MB/sec execute 68 sec latency 50126.040 ms 2024-04-22T22:27:22.396 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:22 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:22.070+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:22.396 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:22 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:22.070+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:22.396 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:22 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:22.070+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:48.618376+0000 front 2024-04-22T22:26:48.618350+0000 (oldest deadline 2024-04-22T22:27:18.619008+0000) 2024-04-22T22:27:22.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:22 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:22.400+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:22.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:22 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:22.400+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:22.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:22 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:22.400+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:23.074 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.82 MB/sec execute 69 sec latency 51126.073 ms 2024-04-22T22:27:23.416 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:23 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:23.084+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:23.416 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:23 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:23.084+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:23.416 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:23 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:23.084+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:48.618376+0000 front 2024-04-22T22:26:48.618350+0000 (oldest deadline 2024-04-22T22:27:18.619008+0000) 2024-04-22T22:27:23.708 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:23 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:23.421+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:23.708 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:23 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:23.421+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:23.708 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:23 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:23.421+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:24.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.77 MB/sec execute 70 sec latency 52126.095 ms 2024-04-22T22:27:24.460 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:24 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:24.110+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:24.460 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:24 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:24.110+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:24.460 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:24 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:24.110+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:48.618376+0000 front 2024-04-22T22:26:48.618350+0000 (oldest deadline 2024-04-22T22:27:18.619008+0000) 2024-04-22T22:27:24.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:24 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:24.465+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:24.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:24 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:24.465+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:24.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:24 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:24.465+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:25.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.72 MB/sec execute 71 sec latency 53126.139 ms 2024-04-22T22:27:25.485 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:25 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:25.155+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:34.616292+0000 front 2024-04-22T22:26:34.616268+0000 (oldest deadline 2024-04-22T22:26:58.716375+0000) 2024-04-22T22:27:25.485 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:25 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:25.155+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:42.717828+0000 front 2024-04-22T22:26:42.717816+0000 (oldest deadline 2024-04-22T22:27:08.617929+0000) 2024-04-22T22:27:25.485 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:25 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:25.155+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:48.618376+0000 front 2024-04-22T22:26:48.618350+0000 (oldest deadline 2024-04-22T22:27:18.619008+0000) 2024-04-22T22:27:25.485 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:25 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6[34279]: 2024-04-22T22:27:25.155+0000 7fc6ec321700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.47:6830 osd.3 since back 2024-04-22T22:26:58.619382+0000 front 2024-04-22T22:26:58.619334+0000 (oldest deadline 2024-04-22T22:27:24.519416+0000) 2024-04-22T22:27:25.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:25 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:25.490+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:25.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:25 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:25.490+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:25.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:25 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:25.490+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:25.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:25 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:25.490+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6830 osd.3 since back 2024-04-22T22:27:00.693918+0000 front 2024-04-22T22:27:00.693932+0000 (oldest deadline 2024-04-22T22:27:24.793968+0000) 2024-04-22T22:27:26.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.66 MB/sec execute 72 sec latency 54126.179 ms 2024-04-22T22:27:26.633 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:26 smithi113 podman[40705]: 2024-04-22 22:27:26.355430061 +0000 UTC m=+5.595355378 container died 87b85a4c560eecad35a770f1f095d3fec1249b1d8d5a5e75261cf8634f4bffae (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.openshift.expose-services=, name=centos-stream, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, ceph=True, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T22:27:26.633 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:26 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:26.452+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:26.633 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:26 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:26.452+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:26.633 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:26 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:26.452+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:26.634 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:26 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:26.452+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6830 osd.3 since back 2024-04-22T22:27:00.693918+0000 front 2024-04-22T22:27:00.693932+0000 (oldest deadline 2024-04-22T22:27:24.793968+0000) 2024-04-22T22:27:27.000 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:26 smithi113 podman[40705]: 2024-04-22 22:27:26.730597992 +0000 UTC m=+5.970523309 container cleanup 87b85a4c560eecad35a770f1f095d3fec1249b1d8d5a5e75261cf8634f4bffae (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6, io.openshift.expose-services=, vcs-type=git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, version=8, io.buildah.version=1.19.8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, name=centos-stream, maintainer=Guillaume Abrioux ) 2024-04-22T22:27:27.000 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:26 smithi113 bash[40705]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6 2024-04-22T22:27:27.000 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:26 smithi113 bash[40736]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-22T22:27:27.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.61 MB/sec execute 73 sec latency 55126.222 ms 2024-04-22T22:27:27.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:27 smithi113 podman[40724]: 2024-04-22 22:27:27.006655349 +0000 UTC m=+0.647325332 container remove 87b85a4c560eecad35a770f1f095d3fec1249b1d8d5a5e75261cf8634f4bffae (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6, architecture=x86_64, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, RELEASE=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , io.openshift.expose-services=, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, release=754, build-date=2022-05-03T08:36:31.336870, ceph=True, version=8, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-22T22:27:27.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:27 smithi113 podman[40801]: 2024-04-22 22:27:27.174017856 +0000 UTC m=+0.111055153 container create 882ff5108d701e13132099cd885386cb6f4858cf1082e8c33282087123f36059 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6-deactivate, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, vcs-type=git, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, version=8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.buildah.version=1.19.8, ceph=True) 2024-04-22T22:27:27.251 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:27 smithi113 podman[40801]: 2024-04-22 22:27:27.082015057 +0000 UTC m=+0.019052367 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:27:27.667 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:27 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:27.487+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:27.668 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:27 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:27.487+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:27.668 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:27 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:27.487+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:27.668 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:27 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:27.487+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6830 osd.3 since back 2024-04-22T22:27:00.693918+0000 front 2024-04-22T22:27:00.693932+0000 (oldest deadline 2024-04-22T22:27:24.793968+0000) 2024-04-22T22:27:27.668 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:27 smithi113 podman[40801]: 2024-04-22 22:27:27.384614608 +0000 UTC m=+0.321651909 container init 882ff5108d701e13132099cd885386cb6f4858cf1082e8c33282087123f36059 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vcs-type=git, RELEASE=HEAD, release=754, distribution-scope=public, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., ceph=True, architecture=x86_64, io.openshift.tags=base centos centos-stream) 2024-04-22T22:27:27.668 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:27 smithi113 podman[40801]: 2024-04-22 22:27:27.387852797 +0000 UTC m=+0.324890095 container start 882ff5108d701e13132099cd885386cb6f4858cf1082e8c33282087123f36059 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6-deactivate, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.openshift.expose-services=, version=8, RELEASE=HEAD, architecture=x86_64, vcs-type=git, ceph=True, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:27:27.669 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:27 smithi113 podman[40801]: 2024-04-22 22:27:27.476173871 +0000 UTC m=+0.413211297 container attach 882ff5108d701e13132099cd885386cb6f4858cf1082e8c33282087123f36059 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vcs-type=git, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, ceph=True, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc.) 2024-04-22T22:27:27.981 INFO:journalctl@ceph.osd.6.smithi113.stdout:Apr 22 22:27:27 smithi113 podman[40801]: 2024-04-22 22:27:27.674777145 +0000 UTC m=+0.611814447 container died 882ff5108d701e13132099cd885386cb6f4858cf1082e8c33282087123f36059 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, vcs-type=git, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, release=754, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, architecture=x86_64, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T22:27:28.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.56 MB/sec execute 74 sec latency 56126.247 ms 2024-04-22T22:27:28.268 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.6.service' 2024-04-22T22:27:28.304 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:27:28.305 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-22T22:27:28.305 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-22T22:27:28.305 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.7 2024-04-22T22:27:28.501 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:28 smithi113 systemd[1]: Stopping Ceph osd.7 for dbcf7660-00f5-11ef-bc93-c7b262605968... 2024-04-22T22:27:28.501 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:28 smithi113 bash[40880]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-22T22:27:28.502 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:28 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:28.490+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:28.502 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:28 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:28.490+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:28.502 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:28 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:28.490+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:28.502 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:28 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:28.490+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6830 osd.3 since back 2024-04-22T22:27:00.693918+0000 front 2024-04-22T22:27:00.693932+0000 (oldest deadline 2024-04-22T22:27:24.793968+0000) 2024-04-22T22:27:29.001 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:28 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:28.540+0000 7f4f9f1c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:27:29.001 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:28 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:28.540+0000 7f4f9f1c1700 -1 osd.7 70 *** Got signal Terminated *** 2024-04-22T22:27:29.002 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:28 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:28.540+0000 7f4f9f1c1700 -1 osd.7 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:27:29.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.52 MB/sec execute 75 sec latency 57126.291 ms 2024-04-22T22:27:30.001 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:29 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:29.537+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:30.001 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:29 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:29.537+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:30.002 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:29 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:29.537+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:30.002 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:29 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:29.537+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6830 osd.3 since back 2024-04-22T22:27:00.693918+0000 front 2024-04-22T22:27:00.693932+0000 (oldest deadline 2024-04-22T22:27:24.793968+0000) 2024-04-22T22:27:30.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.47 MB/sec execute 76 sec latency 58126.335 ms 2024-04-22T22:27:30.790 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:30 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:30.524+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:30.791 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:30 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:30.524+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:30.791 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:30 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:30.524+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:30.791 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:30 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:30.524+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6830 osd.3 since back 2024-04-22T22:27:00.693918+0000 front 2024-04-22T22:27:00.693932+0000 (oldest deadline 2024-04-22T22:27:24.793968+0000) 2024-04-22T22:27:31.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.43 MB/sec execute 77 sec latency 59126.385 ms 2024-04-22T22:27:31.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:31 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:31.481+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:31.752 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:31 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:31.481+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:31.752 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:31 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:31.481+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:31.752 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:31 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:31.481+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6830 osd.3 since back 2024-04-22T22:27:00.693918+0000 front 2024-04-22T22:27:00.693932+0000 (oldest deadline 2024-04-22T22:27:24.793968+0000) 2024-04-22T22:27:32.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.38 MB/sec execute 78 sec latency 60126.413 ms 2024-04-22T22:27:32.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:32 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:32.436+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:32.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:32 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:32.436+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:32.752 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:32 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:32.436+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:32.752 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:32 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:32.436+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6830 osd.3 since back 2024-04-22T22:27:00.693918+0000 front 2024-04-22T22:27:00.693932+0000 (oldest deadline 2024-04-22T22:27:24.793968+0000) 2024-04-22T22:27:33.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.34 MB/sec execute 79 sec latency 61126.458 ms 2024-04-22T22:27:33.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:33 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:33.465+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6806 osd.0 since back 2024-04-22T22:26:35.790456+0000 front 2024-04-22T22:26:35.789555+0000 (oldest deadline 2024-04-22T22:26:59.889449+0000) 2024-04-22T22:27:33.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:33 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:33.465+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6814 osd.1 since back 2024-04-22T22:26:44.298287+0000 front 2024-04-22T22:26:44.292107+0000 (oldest deadline 2024-04-22T22:27:08.992163+0000) 2024-04-22T22:27:33.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:33 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:33.465+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6822 osd.2 since back 2024-04-22T22:26:53.093135+0000 front 2024-04-22T22:26:53.093089+0000 (oldest deadline 2024-04-22T22:27:17.793093+0000) 2024-04-22T22:27:33.752 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:33 smithi113 ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7[37368]: 2024-04-22T22:27:33.465+0000 7f4f9a359700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.47:6830 osd.3 since back 2024-04-22T22:27:00.693918+0000 front 2024-04-22T22:27:00.693932+0000 (oldest deadline 2024-04-22T22:27:24.793968+0000) 2024-04-22T22:27:34.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.30 MB/sec execute 80 sec latency 62126.506 ms 2024-04-22T22:27:34.449 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:34 smithi113 podman[40893]: 2024-04-22 22:27:34.154059114 +0000 UTC m=+5.730882701 container died 8ab7d8d392127a6044e0fbe485c29e38efacfdb9d62e61fde300f04e627bdb78 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, version=8, name=centos-stream, architecture=x86_64, maintainer=Guillaume Abrioux ) 2024-04-22T22:27:34.450 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:34 smithi113 podman[40893]: 2024-04-22 22:27:34.456176796 +0000 UTC m=+6.033000366 container cleanup 8ab7d8d392127a6044e0fbe485c29e38efacfdb9d62e61fde300f04e627bdb78 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, architecture=x86_64, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870) 2024-04-22T22:27:34.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:34 smithi113 bash[40893]: ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7 2024-04-22T22:27:34.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:34 smithi113 bash[40923]: Error: no container with name or ID "ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-22T22:27:34.751 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:34 smithi113 podman[40908]: 2024-04-22 22:27:34.682227899 +0000 UTC m=+0.529614087 container remove 8ab7d8d392127a6044e0fbe485c29e38efacfdb9d62e61fde300f04e627bdb78 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, version=8, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T22:27:35.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.26 MB/sec execute 81 sec latency 63126.550 ms 2024-04-22T22:27:35.179 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:34 smithi113 podman[40988]: 2024-04-22 22:27:34.883191825 +0000 UTC m=+0.109019998 container create 03d49a8ae953db3a28d28c9455b6de3f61bfa89d551037a057dae28aed94ebb1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, vcs-type=git, io.openshift.tags=base centos centos-stream, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_CLEAN=True) 2024-04-22T22:27:35.180 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:34 smithi113 podman[40988]: 2024-04-22 22:27:34.792732954 +0000 UTC m=+0.018561127 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:27:35.180 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:35 smithi113 podman[40988]: 2024-04-22 22:27:35.102258558 +0000 UTC m=+0.328086732 container init 03d49a8ae953db3a28d28c9455b6de3f61bfa89d551037a057dae28aed94ebb1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, version=8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, release=754, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.openshift.expose-services=, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, ceph=True, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-22T22:27:35.180 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:35 smithi113 podman[40988]: 2024-04-22 22:27:35.105521122 +0000 UTC m=+0.331349299 container start 03d49a8ae953db3a28d28c9455b6de3f61bfa89d551037a057dae28aed94ebb1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7-deactivate, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, version=8, distribution-scope=public, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=) 2024-04-22T22:27:35.180 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:35 smithi113 podman[40988]: 2024-04-22 22:27:35.185507677 +0000 UTC m=+0.411335862 container attach 03d49a8ae953db3a28d28c9455b6de3f61bfa89d551037a057dae28aed94ebb1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, release=754, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, ceph=True, vcs-type=git, version=8, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T22:27:35.501 INFO:journalctl@ceph.osd.7.smithi113.stdout:Apr 22 22:27:35 smithi113 podman[40988]: 2024-04-22 22:27:35.384772521 +0000 UTC m=+0.610600688 container died 03d49a8ae953db3a28d28c9455b6de3f61bfa89d551037a057dae28aed94ebb1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbcf7660-00f5-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, vcs-type=git, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, name=centos-stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, release=754) 2024-04-22T22:27:36.000 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbcf7660-00f5-11ef-bc93-c7b262605968@osd.7.service' 2024-04-22T22:27:36.036 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:27:36.036 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-22T22:27:36.037 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T22:27:36.075 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.22 MB/sec execute 82 sec latency 64126.594 ms 2024-04-22T22:27:37.076 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.18 MB/sec execute 83 sec latency 65126.643 ms 2024-04-22T22:27:38.076 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.14 MB/sec execute 84 sec latency 66126.688 ms 2024-04-22T22:27:39.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.10 MB/sec execute 85 sec latency 67128.106 ms 2024-04-22T22:27:40.077 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.07 MB/sec execute 86 sec latency 68128.135 ms 2024-04-22T22:27:41.077 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.03 MB/sec execute 87 sec latency 69128.182 ms 2024-04-22T22:27:42.077 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 3.00 MB/sec execute 88 sec latency 70128.223 ms 2024-04-22T22:27:43.077 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.96 MB/sec execute 89 sec latency 71128.262 ms 2024-04-22T22:27:44.077 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.93 MB/sec execute 90 sec latency 72128.314 ms 2024-04-22T22:27:45.077 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.90 MB/sec execute 91 sec latency 73128.363 ms 2024-04-22T22:27:46.077 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.87 MB/sec execute 92 sec latency 74128.405 ms 2024-04-22T22:27:47.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.84 MB/sec execute 93 sec latency 75128.449 ms 2024-04-22T22:27:48.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.81 MB/sec execute 94 sec latency 76128.764 ms 2024-04-22T22:27:49.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.78 MB/sec execute 95 sec latency 77128.831 ms 2024-04-22T22:27:50.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.75 MB/sec execute 96 sec latency 78128.880 ms 2024-04-22T22:27:51.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.72 MB/sec execute 97 sec latency 79128.915 ms 2024-04-22T22:27:52.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.69 MB/sec execute 98 sec latency 80128.954 ms 2024-04-22T22:27:53.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.66 MB/sec execute 99 sec latency 81128.990 ms 2024-04-22T22:27:54.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.64 MB/sec execute 100 sec latency 82129.027 ms 2024-04-22T22:27:55.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.61 MB/sec execute 101 sec latency 83129.045 ms 2024-04-22T22:27:56.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.59 MB/sec execute 102 sec latency 84129.080 ms 2024-04-22T22:27:57.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.56 MB/sec execute 103 sec latency 85129.097 ms 2024-04-22T22:27:58.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.54 MB/sec execute 104 sec latency 86129.124 ms 2024-04-22T22:27:59.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.51 MB/sec execute 105 sec latency 87129.144 ms 2024-04-22T22:28:00.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.49 MB/sec execute 106 sec latency 88129.191 ms 2024-04-22T22:28:00.971 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T22:28:01.078 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.47 MB/sec execute 107 sec latency 89129.240 ms 2024-04-22T22:28:02.079 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.44 MB/sec execute 108 sec latency 90129.291 ms 2024-04-22T22:28:03.079 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.42 MB/sec execute 109 sec latency 91129.344 ms 2024-04-22T22:28:04.079 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.40 MB/sec execute 110 sec latency 92129.361 ms 2024-04-22T22:28:05.079 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.38 MB/sec execute 111 sec latency 93129.407 ms 2024-04-22T22:28:06.079 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.36 MB/sec execute 112 sec latency 94129.479 ms 2024-04-22T22:28:07.079 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.33 MB/sec execute 113 sec latency 95129.568 ms 2024-04-22T22:28:08.079 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.31 MB/sec execute 114 sec latency 96129.679 ms 2024-04-22T22:28:09.079 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.29 MB/sec execute 115 sec latency 97129.763 ms 2024-04-22T22:28:10.079 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.27 MB/sec execute 116 sec latency 98129.890 ms 2024-04-22T22:28:11.079 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.25 MB/sec execute 117 sec latency 99129.944 ms 2024-04-22T22:28:12.080 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.24 MB/sec execute 118 sec latency 100130.044 ms 2024-04-22T22:28:13.080 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.22 MB/sec execute 119 sec latency 101130.157 ms 2024-04-22T22:28:14.080 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.20 MB/sec execute 120 sec latency 102130.249 ms 2024-04-22T22:28:15.080 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.18 MB/sec execute 121 sec latency 103130.359 ms 2024-04-22T22:28:16.080 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.16 MB/sec execute 122 sec latency 104130.485 ms 2024-04-22T22:28:17.080 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.14 MB/sec execute 123 sec latency 105130.597 ms 2024-04-22T22:28:18.080 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.13 MB/sec execute 124 sec latency 106130.697 ms 2024-04-22T22:28:19.080 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.11 MB/sec execute 125 sec latency 107130.779 ms 2024-04-22T22:28:20.080 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.09 MB/sec execute 126 sec latency 108130.867 ms 2024-04-22T22:28:20.142 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:28:20.175 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:28:20.200 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T22:28:20.202 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847/remote/smithi047/crash 2024-04-22T22:28:20.203 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/crash -- . 2024-04-22T22:28:20.251 INFO:teuthology.orchestra.run.smithi047.stderr:tar: /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T22:28:20.252 INFO:teuthology.orchestra.run.smithi047.stderr:tar: Error is not recoverable: exiting now 2024-04-22T22:28:20.253 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847/remote/smithi113/crash 2024-04-22T22:28:20.254 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/crash -- . 2024-04-22T22:28:20.321 INFO:teuthology.orchestra.run.smithi113.stderr:tar: /var/lib/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T22:28:20.321 INFO:teuthology.orchestra.run.smithi113.stderr:tar: Error is not recoverable: exiting now 2024-04-22T22:28:20.322 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T22:28:20.323 DEBUG:teuthology.orchestra.run.smithi047:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-22T22:28:20.352 INFO:tasks.cephadm:Compressing logs... 2024-04-22T22:28:20.352 DEBUG:teuthology.orchestra.run.smithi047:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T22:28:20.394 DEBUG:teuthology.orchestra.run.smithi113:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T22:28:20.421 INFO:teuthology.orchestra.run.smithi113.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T22:28:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:: No such file or directory 2024-04-22T22:28:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.log 2024-04-22T22:28:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T22:28:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-22T22:28:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T22:28:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi113.rtxmqv.log 2024-04-22T22:28:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.7.log: /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mon.smithi113.log 2024-04-22T22:28:20.422 INFO:teuthology.orchestra.run.smithi113.stderr: 86.6% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T22:28:20.423 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi113.rtxmqv.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T22:28:20.423 INFO:teuthology.orchestra.run.smithi113.stderr: 60.7% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi113.rtxmqv.log.gz 2024-04-22T22:28:20.423 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mon.smithi113.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mgr.smithi113.qvxssq.log 2024-04-22T22:28:20.423 INFO:teuthology.orchestra.run.smithi113.stderr: 90.9% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T22:28:20.423 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-22T22:28:20.423 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mgr.smithi113.qvxssq.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T22:28:20.423 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T22:28:20.423 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.cephadm.log: 81.9% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T22:28:20.425 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-volume.log: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T22:28:20.430 INFO:teuthology.orchestra.run.smithi113.stderr: 92.3% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mgr.smithi113.qvxssq.log.gz 2024-04-22T22:28:20.430 INFO:teuthology.orchestra.run.smithi113.stderr: 90.7% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T22:28:20.431 INFO:teuthology.orchestra.run.smithi047.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T22:28:20.431 INFO:teuthology.orchestra.run.smithi047.stderr:: No such file or directory 2024-04-22T22:28:20.432 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.log 2024-04-22T22:28:20.432 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T22:28:20.432 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T22:28:20.432 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T22:28:20.432 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T22:28:20.432 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T22:28:20.432 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- 86.7% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T22:28:20.432 INFO:teuthology.orchestra.run.smithi047.stderr: /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T22:28:20.433 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mgr.smithi047.uutpnj.log 2024-04-22T22:28:20.433 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi047.avrnmr.log 2024-04-22T22:28:20.433 INFO:teuthology.orchestra.run.smithi047.stderr: 82.9% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T22:28:20.433 INFO:teuthology.orchestra.run.smithi047.stderr: 90.7% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T22:28:20.433 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mgr.smithi047.uutpnj.log: gzip -5 --verbose -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T22:28:20.433 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi047.avrnmr.log: gzip -5 --verbose 80.4% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi047.avrnmr.log.gz 2024-04-22T22:28:20.433 INFO:teuthology.orchestra.run.smithi047.stderr: -- /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mon.smithi047.log 2024-04-22T22:28:20.433 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mon.smithi047.log: 90.8% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T22:28:20.434 INFO:teuthology.orchestra.run.smithi047.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T22:28:20.516 INFO:teuthology.orchestra.run.smithi113.stderr: 92.6% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mon.smithi113.log.gz 2024-04-22T22:28:20.584 INFO:teuthology.orchestra.run.smithi047.stderr: 88.8% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mgr.smithi047.uutpnj.log.gz 2024-04-22T22:28:20.866 INFO:teuthology.orchestra.run.smithi047.stderr: 91.9% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-mon.smithi047.log.gz 2024-04-22T22:28:21.081 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.08 MB/sec execute 127 sec latency 109130.913 ms 2024-04-22T22:28:22.081 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.06 MB/sec execute 128 sec latency 110130.957 ms 2024-04-22T22:28:22.763 INFO:teuthology.orchestra.run.smithi047.stderr: 94.5% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T22:28:23.081 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.04 MB/sec execute 129 sec latency 111130.997 ms 2024-04-22T22:28:24.081 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.03 MB/sec execute 130 sec latency 112131.052 ms 2024-04-22T22:28:24.380 INFO:teuthology.orchestra.run.smithi113.stderr: 94.3% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-22T22:28:24.562 INFO:teuthology.orchestra.run.smithi113.stderr: 94.1% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T22:28:25.081 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.01 MB/sec execute 131 sec latency 113131.103 ms 2024-04-22T22:28:25.119 INFO:teuthology.orchestra.run.smithi047.stderr: 94.2% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T22:28:25.308 INFO:teuthology.orchestra.run.smithi047.stderr: 94.3% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T22:28:25.425 INFO:teuthology.orchestra.run.smithi047.stderr: 94.2% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T22:28:25.427 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-04-22T22:28:25.428 INFO:teuthology.orchestra.run.smithi047.stderr:real 0m5.019s 2024-04-22T22:28:25.428 INFO:teuthology.orchestra.run.smithi047.stderr:user 0m16.945s 2024-04-22T22:28:25.428 INFO:teuthology.orchestra.run.smithi047.stderr:sys 0m0.614s 2024-04-22T22:28:25.592 INFO:teuthology.orchestra.run.smithi113.stderr: 94.3% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T22:28:26.081 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 2.00 MB/sec execute 132 sec latency 114131.180 ms 2024-04-22T22:28:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.98 MB/sec execute 133 sec latency 115131.294 ms 2024-04-22T22:28:27.770 INFO:teuthology.orchestra.run.smithi113.stderr: 94.2% -- replaced with /var/log/ceph/dbcf7660-00f5-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-22T22:28:27.771 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-22T22:28:27.771 INFO:teuthology.orchestra.run.smithi113.stderr:real 0m7.363s 2024-04-22T22:28:27.772 INFO:teuthology.orchestra.run.smithi113.stderr:user 0m19.998s 2024-04-22T22:28:27.772 INFO:teuthology.orchestra.run.smithi113.stderr:sys 0m0.732s 2024-04-22T22:28:27.772 INFO:tasks.cephadm:Archiving logs... 2024-04-22T22:28:27.772 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847/remote/smithi047/log 2024-04-22T22:28:27.773 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T22:28:28.104 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.97 MB/sec execute 134 sec latency 116131.382 ms 2024-04-22T22:28:29.099 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.95 MB/sec execute 135 sec latency 117131.466 ms 2024-04-22T22:28:30.063 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847/remote/smithi113/log 2024-04-22T22:28:30.064 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T22:28:30.081 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.94 MB/sec execute 136 sec latency 118131.587 ms 2024-04-22T22:28:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.93 MB/sec execute 137 sec latency 119131.685 ms 2024-04-22T22:28:32.103 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.91 MB/sec execute 138 sec latency 120131.791 ms 2024-04-22T22:28:32.970 INFO:tasks.cephadm:Removing cluster... 2024-04-22T22:28:32.970 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 --force 2024-04-22T22:28:33.082 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.90 MB/sec execute 139 sec latency 121131.858 ms 2024-04-22T22:28:33.282 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 --force 2024-04-22T22:28:33.585 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T22:28:33.585 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T22:28:33.601 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T22:28:33.616 INFO:tasks.cephadm:Teardown complete 2024-04-22T22:28:33.616 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T22:28:33.626 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T22:28:33.627 DEBUG:teuthology.orchestra.run.smithi047:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T22:28:33.643 DEBUG:teuthology.orchestra.run.smithi113:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T22:28:33.656 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: ntpq: command not found 2024-04-22T22:28:33.659 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T22:28:33.659 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-04-22T22:28:33.660 INFO:teuthology.orchestra.run.smithi047.stdout:^- hv01.front.sepia.ceph.com 2 6 377 117 +793us[ +790us] +/- 594ms 2024-04-22T22:28:33.660 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 52 -124us[ -124us] +/- 77ms 2024-04-22T22:28:33.660 INFO:teuthology.orchestra.run.smithi047.stdout:^* hv03.front.sepia.ceph.com 3 6 377 56 +86us[ +83us] +/- 72ms 2024-04-22T22:28:33.660 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:28:33.669 INFO:teuthology.orchestra.run.smithi113.stderr:bash: line 1: ntpq: command not found 2024-04-22T22:28:33.673 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T22:28:33.673 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-04-22T22:28:33.673 INFO:teuthology.orchestra.run.smithi113.stdout:^- hv01.front.sepia.ceph.com 2 6 377 51 +774us[ +774us] +/- 595ms 2024-04-22T22:28:33.673 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 -114us[ -129us] +/- 77ms 2024-04-22T22:28:33.674 INFO:teuthology.orchestra.run.smithi113.stdout:^* hv03.front.sepia.ceph.com 3 6 377 52 +91us[ +75us] +/- 72ms 2024-04-22T22:28:33.674 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:28:33.675 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T22:28:33.684 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T22:28:33.685 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T22:28:33.694 DEBUG:teuthology.orchestra.run.smithi047:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T22:28:33.772 DEBUG:teuthology.orchestra.run.smithi113:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T22:28:33.833 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T22:28:33.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:28:33.860 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T22:28:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:28:33.886 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T22:28:33.897 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T22:28:33.942 INFO:teuthology.task.internal:Duration was 988.915894 seconds 2024-04-22T22:28:33.942 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T22:28:33.952 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T22:28:33.952 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T22:28:33.955 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T22:28:33.990 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T22:28:33.991 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T22:28:34.082 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.88 MB/sec execute 140 sec latency 122131.973 ms 2024-04-22T22:28:34.438 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T22:28:34.438 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi047.front.sepia.ceph.com 2024-04-22T22:28:34.439 DEBUG:teuthology.orchestra.run.smithi047:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T22:28:34.499 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi113.front.sepia.ceph.com 2024-04-22T22:28:34.499 DEBUG:teuthology.orchestra.run.smithi113:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T22:28:34.526 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T22:28:34.526 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T22:28:34.542 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T22:28:34.592 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T22:28:34.592 DEBUG:teuthology.orchestra.run.smithi047:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T22:28:34.609 DEBUG:teuthology.orchestra.run.smithi113:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T22:28:35.082 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.87 MB/sec execute 141 sec latency 123132.001 ms 2024-04-22T22:28:36.082 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.86 MB/sec execute 142 sec latency 124132.029 ms 2024-04-22T22:28:37.082 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.84 MB/sec execute 143 sec latency 125132.078 ms 2024-04-22T22:28:38.082 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.83 MB/sec execute 144 sec latency 126132.112 ms 2024-04-22T22:28:39.082 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.82 MB/sec execute 145 sec latency 127132.140 ms 2024-04-22T22:28:39.656 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T22:28:39.667 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T22:28:39.668 DEBUG:teuthology.orchestra.run.smithi047:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T22:28:39.692 DEBUG:teuthology.orchestra.run.smithi113:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T22:28:39.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T22:28:39.732 DEBUG:teuthology.orchestra.run.smithi047:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T22:28:39.734 DEBUG:teuthology.orchestra.run.smithi113:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T22:28:39.754 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = core 2024-04-22T22:28:39.787 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = core 2024-04-22T22:28:39.811 DEBUG:teuthology.orchestra.run.smithi047:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T22:28:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:28:39.841 DEBUG:teuthology.orchestra.run.smithi113:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T22:28:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:28:39.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T22:28:39.878 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T22:28:39.879 DEBUG:teuthology.misc:Transferring archived files from smithi047:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847/remote/smithi047 2024-04-22T22:28:39.879 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T22:28:39.938 DEBUG:teuthology.misc:Transferring archived files from smithi113:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847/remote/smithi113 2024-04-22T22:28:39.939 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T22:28:39.981 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T22:28:39.981 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T22:28:39.984 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T22:28:40.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T22:28:40.042 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T22:28:40.043 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T22:28:40.076 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T22:28:40.076 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T22:28:40.079 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T22:28:40.082 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.81 MB/sec execute 146 sec latency 128132.164 ms 2024-04-22T22:28:40.091 INFO:teuthology.orchestra.run.smithi047.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 22:28 /home/ubuntu/cephtest 2024-04-22T22:28:40.094 INFO:teuthology.orchestra.run.smithi113.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 22:28 /home/ubuntu/cephtest 2024-04-22T22:28:40.095 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T22:28:40.106 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T22:28:40.128 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T22:28:40.157 DEBUG:teuthology.nuke:shortname: smithi047 2024-04-22T22:28:40.158 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T22:28:40.179 DEBUG:teuthology.nuke:shortname: smithi113 2024-04-22T22:28:40.179 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T22:28:40.183 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 22:05:51.277827', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLACz67ci4U+6RaRmz34a4uSK2gsOM9VQI/yt1dsB7TAg6+kAmT2vpjOzwwksI0Rn331pohypAOrGQD6YegSLDk='} 2024-04-22T22:28:40.204 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668847', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 22:05:51.276490', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCFuda/Xe+k4NML1BaUWlkaIhBmK2uzlS+3ShjPyv66TxnQL0e31CxtKAbjMJXKZ2ROKJX8y6D0TwL6fbYWF/Zc='} 2024-04-22T22:28:40.240 INFO:teuthology.orchestra.console.smithi047:Power off 2024-04-22T22:28:40.241 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T22:28:40.259 INFO:teuthology.orchestra.console.smithi113:Power off 2024-04-22T22:28:40.259 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T22:28:40.270 DEBUG:teuthology.orchestra.console.smithi047:power off output: Chassis Power Control: Down/Off 2024-04-22T22:28:40.270 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:28:40.283 DEBUG:teuthology.orchestra.console.smithi113:power off output: Chassis Power Control: Down/Off 2024-04-22T22:28:40.283 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:28:40.296 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-04-22T22:28:40.312 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-04-22T22:28:41.083 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.79 MB/sec execute 147 sec latency 129132.267 ms 2024-04-22T22:28:42.082 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.78 MB/sec execute 148 sec latency 130132.405 ms 2024-04-22T22:28:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.77 MB/sec execute 149 sec latency 131132.491 ms 2024-04-22T22:28:44.083 INFO:teuthology.orchestra.run.smithi047.stdout: 5 83444 1.76 MB/sec execute 150 sec latency 132132.567 ms 2024-04-22T22:28:44.297 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:28:44.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:28:44.313 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:28:44.409 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-04-22T22:28:44.423 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-04-22T22:28:48.414 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:28:48.425 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:28:48.528 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-04-22T22:28:48.535 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is off 2024-04-22T22:28:48.636 INFO:teuthology.orchestra.console.smithi113:Power off completed 2024-04-22T22:28:52.534 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:28:52.645 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is off 2024-04-22T22:28:52.746 INFO:teuthology.orchestra.console.smithi047:Power off completed 2024-04-22T22:28:52.851 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 988.9158937931061 failure_reason: 'Command failed on smithi047 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbcf7660-00f5-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \''ceph versions | jq -e \''"\''"\''.overall | keys\''"\''"\'' | grep $sha1\''''' owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4aaff49bb1d84fe084ea6fecb008f356 status: fail success: false 2024-04-22T22:28:52.852 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T22:28:52.938 INFO:teuthology.run:FAIL